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I.  INTRODUCTION 


The  2008  Commander’s  Guidance  from  Commander,  Naval  Meteorology  and 
Oceanography  Command  (CNMOC)  indicates  that  increased  budget  and  research 
resources  will  be  devoted  to  the  continued  development  of  probabilistic  forecasting 
techniques  and  products.  Deterministic  forecasts  provide  single  values  without  indication 
of  uncertainty  or  reliability,  while  probabilistic  forecasts  can  help  improve  decision 
making  and  combat  capability  by  indicating  ranges  of  possibilities  (Eckel  2008).  Prior 
research  demonstrates  the  advantages  of  stochastic  forecasting  in  the  context  of  a  series 
of  repeated,  independent  decisions  (Thompson  1952,  Eckel  2008).  The  focus  of  this 
study  is  to  demonstrate  the  net  benefits  of  stochastic  forecast-based  decision  making  in  a 
complex,  multi-step  weather  impacts  simulator  of  Operation  IRAQI  FREEDOM  (OIF), 
the  Weather  Impact  Assessment  Tool  (WIAT). 

The  main  objectives  are  to:  1)  demonstrate  improvement  to  simulated  campaign 
operations  by  utilizing  probabilistic  forecasting,  and  2)  investigate  the  existence  of 
optimal  decision  probability  thresholds  for  weather  parameters  in  the  simulated 
campaign.  The  main  benefit  of  this  study  will  be  the  application  of  stochastic  forecasting 
to  operational  risk  management  (ORM)  in  the  context  of  complex  operational  decision 
making  in  a  simulated  strike  warfare  campaign.  It  will  also  provide  an  example  of  how 
probabilistic  forecasting  can  be  incorporated  into  a  variety  of  similar  warfare  types 
beyond  strike  warfare. 

This  thesis  is  organized  into  five  chapters.  In  Chapter  II,  we  discuss  the  rationale 
for  developing  stochastic  forecasts.  Chapter  III  describes  the  WIAT  program.  Chapter  IV 
presents  the  methodology  for  running  WIAT  in  deterministic  and  stochastic  modes  to 
meet  the  above  objectives.  Chapter  V  shows  output  metrics  and  compares  the  results  of 
campaigns  run  under  detenninistic  and  stochastic  frameworks.  Conclusions  and 
recommendations  are  presented  in  Chapter  VI. 
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II.  BACKGROUND 


A.  SCOPE 

This  chapter  will  discuss  ensemble  forecasting  and  how  operational  risk 
management  relates  to  forecasting.  An  explanation  of  the  sources  of  error  in  numerical 
weather  prediction  (NWP),  deterministic  and  stochastic  prediction  methods,  and 
characteristics  of  ensemble  prediction  systems  (EPS)  is  fundamental  to  understanding  the 
benefit  of  including  forecast  uncertainty  information  in  the  decision  process.  Proper 
application  of  ensemble  forecasts  requires  an  assessment  of  costs,  risks,  and  rewards 
associated  with  employing  certain  units  or  executing  missions. 

B.  ATMOSPHERIC  PREDICTION 

1.  Uncertainty 

Lorenz  (1963)  found  that  small  adjustments  to  the  initial  state  of  a  dynamical 
system  led  to  considerably  different  results  as  time  progressed.  Based  on  this 
experimentation,  he  suggested  that  error  in  describing  the  current  state  of  the  atmosphere 
is  the  major  contributor  to  difficulties  in  predictability.  This  analysis  of  the  atmospheric 
state  is  also  known  as  the  atmosphere’s  initial  conditions  (ICs),  a  description  of  the 
values  associated  with  the  state  of  the  atmosphere  at  a  given  point  in  time.  However,  the 
analysis  ICs  will  never  be  precisely  the  same  as  the  actual  atmospheric  ICs.  Error  in 
determining  analysis  ICs  can  result  from  imperfect  measurements,  from  the  sparse  and 
uneven  distribution  of  atmospheric  observations,  and  from  assumptions  and 
approximations  used  in  assimilating  the  observations.  Limited  model  resolution  also 
adversely  affects  data  assimilation,  and  in  turn,  the  ICs  (Kalnay  2004). 

The  earth’s  atmosphere  is  a  dynamical  system,  and  accordingly  should  be 
predictable  provided  that  its  current  state  and  the  rules  for  its  evolution  can  be  completely 
deduced.  After  determining  the  ICs  for  the  atmosphere,  operational  models  employ  a  set 
of  equations  that  approximate  relationships  among  various  atmospheric  parameters. 
However,  there  are  several  pathways  for  error  to  be  introduced  into  the  model.  The 
differential  equations  that  govern  the  development  of  atmospheric  motion  over  time  do 
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not  completely  describe  all  motion  within  the  atmosphere.  Difficulties  in  modeling  both 
upper  and  lower  boundaries  of  the  atmosphere,  approximations  made  in  the  numerical 
solution  techniques,  various  simplifications  of  small-scale  processes,  and  finite  computer 
precision  all  contribute  error,  so  that  numerical  models  cannot  exactly  project  future 
states  of  the  atmosphere  (Eckel  2008). 

Because  both  the  analysis  and  the  model  are  flawed,  error  introduced  by  the 
analysis  is  magnified  by  forecast  error,  and  the  difference  between  the  actual  state  and  the 
model  state  grows  with  increasing  lead  time.  Even  the  finest  model  resolution  available 
will  not  prevent  the  compounding  of  small  errors  over  time,  and  the  corresponding 
divergence  of  modeled  weather  from  true  weather. 

As  a  result  of  the  increasing  divergence  of  true  weather  and  modeled  weather, 
prediction  capability  diminishes  with  increasing  lead  time.  The  limit  of  predictability  for 
a  specific  type  of  forecast  (a  given  atmospheric  variable)  occurs  at  the  lead  time  where 
the  mean  squared  error  of  the  model  forecast  equals  the  mean  squared  error  of  a  forecast 
of  the  climatological  mean  (Eckel  2008).  Beyond  this  lead  time,  the  operator  receives 
better  forecast  information  from  climatology  than  a  model  forecast. 

2.  Deterministic  and  Stochastic  Prediction  Methods 

The  method  most  commonly  employed  for  atmospheric  weather  prediction  is  the 
deterministic  method,  characterized  by  a  single -value  forecast  or  descriptive  solution. 
The  great  strides  made  in  both  computing  power  and  model  refinement  over  the  last  few 
decades  have  extended  predictability,  but  even  so,  short-  and  medium-term  forecast 
errors  are  occasionally  large  due  to  one  or  more  of  the  sources  of  error  discussed 
previously.  One  example  of  such  a  scenario  was  the  over-forecasted  Thanksgiving  Storm 
of  2001.  The  24-hour  forecast  from  the  Fifth-Generation  National  Center  for 
Atmospheric  Research  /  Penn  State  University  Mesoscale  Model  (MM5)  for  22 
November  2001  (Figure  1)  depicted  an  operational  forecast  for  gale  force  winds  in  the 
Puget  Sound  and  Straits  of  Juan  de  Fuca  (Eckel  2008).  Observations  of  that  period, 
however,  indicated  calm  weather.  Later  analysis  of  the  model,  shown  in  Figure  2, 
indicated  that  the  operational  forecast  had  depicted  a  plausible  but  low-probability  event, 

given  multiple  re-runs  of  the  model;  in  other  words,  the  model  had  forecast  a  possible 
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event,  but  the  forecasters  had  no  indication  of  the  low  likelihood  of  this  forecast. 
Ultimately,  deterministic  forecasts  greatly  limit  the  weather  forecaster’s  ability  to  convey 
valuable  information  about  uncertainty  and  multiple  possible  events.  This  leads  to 
weather  forecast  products  of  both  numeric  and  graphical  types  that  convey  a  false  sense 
of  confidence  to  the  user,  which  negatively  affects  the  quality  of  decision  making  by 
operational  users. 


Figure  1.  MM5  Operational  24-hour  Forecast,  depicting  a  gale  force  wind  event  in  the 
Puget  Sound  and  Straits  of  Juan  de  Fuca.  Valid  1800Z,  22  November  2001. 

From  (Eckel,  2008). 
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Figure  2.  MM5  Verification  Analysis  for  1800Z,  22  November  2001,  depicting  a 
weaker  low-pressure  area  than  forecast,  with  no  gale  force  winds.  From  (Eckel, 

2008). 

Stochastic  forecasting,  on  the  other  hand,  provides  a  suite  of  solutions  that 
objectively  convey  potential  outcomes  and  forecast  uncertainty.  Such  information  allows 
the  user  to  grasp  the  range  of  weather  conditions  (evolved  from  the  present  state)  that 
may  impact  his  operations.  If  the  user  has  constraints  in  operations  that  would  be  broken 
by  one  or  several  potential  outcomes,  a  stochastic  approach  permits  a  better  decision¬ 
making  process  by  providing  a  more  complete  set  of  information.  Instead  of  focusing 
solely  on  limiting  the  forecast  error,  stochastic  forecasting  attempts  to  model  the  flow- 
dependent  error  growth,  and  then  depict  the  uncertainty  in  a  given  forecast  (Eckel  2008). 
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C.  ENSEMBLE  FORECASTING 

1.  Description 

While  there  is  no  standard  means  of  deriving  an  ensemble  forecast,  the  most 
general  method  involves  repeatedly  adding  small  perturbations  to  both  analysis  and 
model  to  produce  multiple  unique  forecast  solutions.  Zhu  et  al.  (2001)  state  that  the 
operational  ensemble  forecasts  produced  by  multiple  prediction  centers  around  the  world 
use  “models... integrated  a  number  of  times,  started  from  slightly  perturbed  initial 
conditions,  in  addition  to  generating  the  traditional  ‘control’  forecast  that  starts  from  the 
best  available  atmospheric  analysis.”  This  incorporation  of  slightly  perturbed  initial 
conditions  (ICs)  simulates  the  analysis  error  inherent  in  portraying  the  atmosphere.  Each 
different  IC  may  then  be  funneled  into  a  slightly  different  version  of  the  given  NWP 
model,  as  a  means  of  accounting  for  model  error.  Systems  that  use  only  one  version  of 
the  model  (an  unperturbed  model)  effectively  neglect  the  model  as  a  source  of  error  and 
uncertainty  in  the  ensemble.  Taken  all  together,  the  ensemble  members  are  used  to 
represent  a  spectrum  of  potential  future  states  of  the  atmosphere.  This  spectrum  provides 
estimates  of  forecast  uncertainty  and  probabilities  for  specific  events.  Each  member 
should  have  an  equal  chance  of  being  the  true  outcome;  incorporating  more  ensemble 
members  (model  runs)  gives  the  ensemble  user  a  better  chance  of  at  least  one  member 
being  close  to  the  truth. 

One  of  the  meteorological  prediction  centers  responsible  for  producing  ensemble 
forecasts  is  the  Fleet  Numerical  Meteorology  and  Oceanography  Center  (FNMOC),  in 
Monterey,  California.  FNMOC  produces  a  number  of  ensemble  products  based  on  the 
Navy  Operational  Global  Atmospheric  Prediction  System  (NOGAPS)  model,  an  example 
of  which  is  shown  in  Figure  3.  The  ensemble  products,  instead  of  displaying  contours  of 
atmospheric  parameters  such  as  pressure,  winds,  and  temperature,  as  shown  in  the  MM5 
graphics  previously,  display  contours  of  probabilities  of  exceeding  a  given 
meteorological  threshold.  In  the  case  of  Figure  3,  the  contours  are  of  the  likelihood  of 
exceeding  6.25  mm  precipitation  in  the  twelve-hour  period  ending  at 
0000Z  13  September  2008. 
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Figure  3.  FNMOC  EFS  Graphic  displaying  96-hour  Ensemble  Forecast  sea  level 
pressure  (contours)  and  probabilities  of  precipitation  exceeding  6.25  mm  over 
the  period  from  1200Z  12  September  2008  to  0000Z  13  September  2008.  From 

FNMOC  Web  site. 


Both  deterministic  and  stochastic  techniques  employ  atmospheric  observations, 
analyses,  modeling,  and  forecasts,  in  the  effort  to  describe  the  atmosphere.  Stochastic 
forecasts,  however,  portray  information  as  probabilities  or  as  probability  density 
functions  (PDFs).  Ensemble  forecasting  pursues  the  production  of  a  “sharp,  calibrated 
forecast  probability  density  function  (PDF)  of  possible  future  states  of  the  atmosphere 
from  which  the  true  state  is  consistently  a  random  sample”  (Eckel  2008). 

The  evolution  of  the  atmosphere  is  fundamentally  a  deterministic  process.  In 

principle,  there  is  an  exact  set  of  characteristics  that  describe  the  atmosphere  at  any  given 

time,  and  a  set  of  rules  that  comprehensively  governs  the  evolution  of  the  atmosphere 

towards  one  specific  state  at  a  given  future  time.  In  practice,  this  analysis  and  evolution 

cannot  be  known  exactly,  but  we  can  construct  a  system  that  will  provide  a  good  sense  of 
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the  likely  analyses  and  changes  over  time.  A  likely  state  of  the  atmosphere  could  be 
represented  as  a  point  in  a  PDF  that  includes  all  possible  states  of  the  atmosphere  at  that 
time  (a  true  analysis  PDF).  A  possible  future  state  of  the  atmosphere  could  also  be 
represented  as  a  point  on  a  PDF  that  includes  all  potential  future  states  (a  true  forecast 
PDF).  Hence,  the  ensemble  forecast  process  pursues  the  production  of  an  analysis  PDF 
that  closely  depicts  the  true  analysis  PDF,  and  a  forecast  PDF  that  approximates  the  true 
forecast  PDF.  Neither  the  analysis  PDF  nor  the  forecast  PDF  are  absolute,  as  they  are 
dependent  on  the  forecast  system  used  to  derive  them;  for  example,  if  more  observations 
are  incorporated  into  the  forecast  system,  the  shape  of  the  analysis  and  forecast  PDFs  will 
change.  In  other  words,  the  ensemble  forecast  process  attempts  to  approximate  a 
hypothetical  distribution  of  weather  states  from  which  the  truth  is  always  a  random 
sample. 


2.  Calibration  and  Ensemble  Size 

Several  problems  are  inherent  to  the  ensemble’s  forecast  PDF.  In  an  ideal 
ensemble,  the  slightly  perturbed  analyses  and  models  incorporate  all  sources  of 
uncertainty;  in  reality,  ensembles  can  only  include  the  primary  sources  of  uncertainty, 
and  only  to  a  limited  degree.  Such  limitations  lead  to  forecast  PDFs,  which  differ 
significantly  from  the  true  forecast  PDFs,  and  thus  forecast  probabilities  (interpreted  as 
forecast  frequencies  of  occurrence)  do  not  agree  with  observed  frequencies  of  occurrence. 
These  forecast  PDFs  are  said  to  be  uncalibrated  or  poorly  calibrated  ensemble  forecasts. 
A  calibrated  forecast  has  been  adjusted  so  that  the  estimated  probability  of  an  event 
matches  the  rate  of  occurrence  of  the  event,  i.e.,  the  forecast  PDF  is  adjusted  so  that  it 
approaches  the  true  forecast  PDF.  For  example,  an  ensemble  initiated  with  a  given 
meteorological  state  returns  a  40%  probability  of  an  event  occurring  at  some  point  in  the 
future.  We  should  expect  that  if  the  same  meteorological  situation  occurs  100  times  in 
nature,  the  forecast  event  would  occur  40  times.  If  instead,  the  event  occurred  30  times, 
it  becomes  apparent  that  the  method  used  in  forecasting  the  event  overestimates  the 
probability  of  occurrence.  Even  if  we  cannot  find  and  correct  the  cause  of  the 
overestimate,  we  can  calibrate  by  adjusting  the  40%  forecast  to  30%. 
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Typically,  narrower  PDFs  indicate  confidence  in  a  smaller  range  of  potential 
outcomes,  while  wider  forecast  PDFs  indicate  more  uncertainty  in  the  forecast.  For 
example,  a  forecaster  is  interested  in  predicting  500-hectopascal  (hPa)  geopotential 
heights  for  a  given  location.  Provided  that  a  good  analysis  is  made,  the  forecaster  can 
expect  to  see  a  fairly  narrow  range  of  potential  outcomes  for  a  12-hour  forecast,  a 
relatively  short  lead  time.  With  increasing  lead  times,  however,  there  is  greater 
opportunity  for  analysis  errors  and  model  errors  to  interact  and  compound  each  other;  for 
a  48-hour  forecast  of  500-hPa  geopotential  heights,  the  forecaster  would  expect  to  see  a 
larger  range  of  values  as  potential  solutions.  This  reflects  the  increased  uncertainty  with 
increasing  lead  time,  and  accordingly  wider  PDFs. 

Finally,  an  ideal  ensemble  would  incorporate  an  infinite  number  of  members  that 
could  “fill  out”  the  forecast  PDF.  With  an  infinite  number  of  members,  all  potential 
forecast  states  would  be  represented,  and  a  complete  distribution  could  be  constructed. 
Instead,  EPSs  sample  the  forecast  PDF  a  finite  number  of  times.  Incomplete  sampling  of 
the  forecast  PDF  introduces  an  additional  source  of  error,  in  that  the  EPS  will  not 
accurately  depict  the  range  of  values  in  a  multi-dimensional  PDF  (pockets  of  values  may 
be  over-  or  under-represented).  In  order  to  achieve  forecast  PDFs  that  approximate  the 
true  forecast  PDFs,  most  EPSs  incorporate  20-30  model  runs  to  ensure  a  skilled  forecast, 
with  upwards  of  50  runs  if  the  system  is  particularly  designed  to  capture  low  probability 
events  found  at  the  tails  of  the  forecast  PDF  (Eckel  2008). 

D.  COST-LOSS  AND  VALUE  ANALYSIS 

Brier  (1944)  was  among  the  first  to  suggest  using  probabilities  of  given  classes  of 
weather  events  in  creating  a  forecast  instead  of  simply  forecasting  one  particular  class  of 
event.  Brier  used  an  example  of  a  movie  producer’s  concern  with  stratus  cover  during  a 
photographic  shoot  as  a  means  to  introduce  a  cost-loss  scenario.  In  his  example,  the 
producer  can  complete  his  work  if  the  skies  are  clear  after  1:00  PM,  and  the  most  likely 
forecast  time  for  cloud  dissipation  is  11:30  AM,  with  some  possibility  that  this  forecast 
may  be  wrong  by  several  hours.  The  producer  is  then  interested  in  the  probability,  P,  of 
this  cloud  dissipation  before  1:00  PM.  The  cost  to  the  producer  to  hire  help,  set  up 

equipment,  etc.  is  the  stake  (S)  put  up  to  achieve  a  gain  (G)  realized  by  completing  the 
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work  that  day.  If  this  decision  were  repeated  over  multiple  trials  (“in  the  long  run”),  the 
producer  could  expect  to  accumulate  a  net  benefit  if  P  >  S/G.  This  means  that  the 
producer  should  only  risk  the  cost  associated  with  setting  up  the  shooting  event  when  the 
probability  of  clear  skies  exceeds  the  ratio  of  the  stake  to  the  prospective  gain  from 
holding  the  event. 

In  Brier’s  example,  the  stake  is  $50,000  and  the  gain  is  $100,000.  The  probability 
assigned  to  the  chance  of  clear  skies  is  0.8.  If  ten  trials  were  held,  and  the  forecast 
probability  was  calibrated  (outcomes  matched  the  probabilities),  then  the  producer  would 
see  a  total  stake  expenditure  of  $500,000  (10  trials  multiplied  by  the  $50,000  stake  per 
trial)  and  a  gain  of  $800,000  (eight  trials  where  the  sky  was  clear  multiplied  by  the 
$100,000  gain  per  successful  trial),  for  a  net  benefit  of  $300,000.  However,  if  the 
probability  assigned  to  the  chance  of  clear  skies  is  calibrated  at  0.4,  the  producer  could 
expect  to  see  a  negative  return  over  the  series  of  decisions  (for  ten  trials,  -$100,000). 
Both  of  these  scenarios  assume  that  the  producer  will  put  forward  the  stake  in  every  trial. 
However,  if  the  producer  understands  that  the  chance  of  clear  skies  is  less  than  the  stake- 
gain  ratio  (in  this  case,  0.5),  then  he  will  decide  not  to  expend  setup  money  for  any  of  the 
trials,  and  accept  a  net  benefit  of  0  rather  than  the  net  loss  of  $100,000.  The  stake-gain 
ratio  becomes  the  producer’s  break-even  point  and  consequently  the  decision  threshold 
for  conducting  the  shoot.  This  sort  of  threshold  is  integral  to  the  campaign  decision¬ 
making  process  both  in  this  study’s  use  of  WIAT  and  in  real-world  scenarios,  as  it  can  be 
utilized  to  evaluate  the  net  benefit  of  conducting  an  operation  within  a  campaign,  as  well 
as  finding  the  threshold  where  benefit  is  optimized  over  a  longer  time  frame. 

Thompson  (1950)  used  a  hypothetical  case  study  to  demonstrate  the  ability  to 
minimize  monetary  costs  taken  for  protective  measures  by  using  a  cost-loss  comparison 
with  the  probability  of  a  given  meteorological  event.  His  posited  construction  company 
was  responsible  for  pouring  concrete  in  Los  Angeles  over  the  October  1949-March  1950 
period.  He  assigned  a  cost  of  $400  to  protect  concrete  from  a  precipitation  event,  while 
the  loss  associated  with  not  protecting  the  concrete  (given  the  occurrence  of  0.15  inch  of 
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rain  within  36  hours  of  the  time  the  concrete  was  poured)  of  $5000.  He  conducted  a 
comparison  of  the  resulting  expenditures  according  to  six  different  strategies  that  could 
have  been  undertaken  by  the  company: 

(1)  receiving  no  forecast  at  all, 

(2)  receiving  no  forecast  but  taking  protective  measures  daily, 

(3)  employing  protective  measures  only  when  climatological  expectancy 
exceeded  the  cost-loss  ratio  (equal  to  0.08), 

(4)  using  persistence  (employing  protective  measures  on  all  days  following  a 
day  with  measurable  precipitation), 

(5)  using  a  categorical  forecast  based  on  whether  the  probability  of  the  rain 
event  was  greater  or  less  than  50%, 

(6)  or  using  a  forecast  based  on  whether  or  not  the  probability  of  the  rain 
event  exceeded  the  cost-loss  ratio  (Thompson  1950). 

The  last  of  these  options  was  found  to  yield  the  lowest  total  expenditure,  a  full  33%  less 
than  the  categorical  forecast. 

The  case  studies  presented  by  Thompson  and  Brier  effectively  illustrate  two 
different  means  of  using  probabilities  to  improve  outcomes  for  the  user.  Brier’s  movie 
producer  is  working  in  a  cost-gain  framework,  where  the  object  is  to  maximize  benefit  (in 
this  case,  return  on  investment  in  the  form  of  completed  photographic  shoots). 
Thompson’s  construction  company  is  operating  in  a  cost-loss  framework,  where  the 
object  is  to  minimize  loss  (payment  to  provide  protective  measures  in  case  of 
thunders tonns).  Both  case  studies,  however,  demonstrate  the  balancing  act  required  in 
accepting  an  appropriate  amount  of  risk  as  compared  to  the  user’s  specific  risk  tolerance, 
as  a  means  to  optimize  long-term  perfonnance. 

While  Brier  speculated  on  a  specific  either/or  scenario  used  to  protect  the  budget 
of  a  movie  producer,  Thompson  expanded  the  list  of  options  available  to  a  decision 
maker.  Comparing  consequences  from  following  various  sources  of  forecast  infonnation, 
the  tailored  forecast  information  based  on  the  company’s  cost-loss  ratio  would  have 
yielded  the  lowest  expenditure  for  the  company.  Thompson  interpreted  these  results  to 
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show  that  probability  estimates  had  advantages,  and  that  detenninistic  forecasts  contained 
an  “inherent  danger”  due  to  the  fact  that  “the  user  is  not  provided  with,  or  does  not  make 
use  of  a  measure  of  the  reliability  of  the  prediction.” 

Zhu  et  al.  (2002)  evaluated  the  economic  value  derived  from  the  use  of  a  14- 
member  ensemble  of  the  Medium  Range  Forecast  (MRF)  T62  model  (referring  to  global 
spectral  model  that  resolves  62  zonal  wave  numbers),  as  compared  to  a  single  T62  run 
and  a  single  T126  run.  A  forecast  has  economic  value  if  the  user  does  better  using  the 
forecast  than  using  a  climatological  forecast;  the  writers’  experiment  was  designed  to 
determine  which  of  the  deterministic  and  stochastic  forecasts  would  most  out-perform  the 
climatological  forecast.  While  the  authors  did  not  specify  a  specific  real-world  cost-loss 
decision,  reducing  the  cumulative  cost  of  protective  measures  characterized  the 
discussion,  much  as  it  did  in  Thompson’s  and  Brier’s  work. 

Zhu  et  al.  found  that  for  prediction  of  500-hPa  heights  over  Northern  Hemisphere 
extratropics,  all  users  were  better  off  with  the  ensemble  forecasts  after  120  hours,  with 
the  vast  majority  better  off  with  ensemble  forecasts  after  72  hours.  The  authors  found 
that  for  cost-loss  ratios  outside  of  a  band  from  0.2  to  0.5,  the  ensemble  forecasts  provided 
better  infonnation  than  either  of  the  control  forecasts.  They  also  demonstrated  that  the 
range  of  cost-loss  ratios  for  which  forecasts  demonstrated  value  was  substantially  wider 
with  the  use  of  ensemble  forecasts.  Since  the  ensemble  of  14  T62  runs  represented 
roughly  the  same  computational  cost  as  the  higher-resolution  T126  single  run,  the 
findings  suggest  that  for  certain  situations,  ensemble  runs  of  a  lower-resolution  model 
can  be  more  cost-effective  than  a  higher-resolution  model  run. 

The  authors  state  that  ensemble-based  distributions  can  exceed  the  performance  of 
the  single  control  run  partly  due  to  the  nonlinear  filtering  that  takes  place  when  fonning 
the  ensemble  mean,  but  more  due  to  its  ability  to  reflect  time  and  space  (flow-dependent) 
variations  in  forecast  uncertainty  (Zhu  et  al.  2002).  The  deterministic  forecasts  had  more 
economic  value  than  the  ensemble  at  lead  times  shorter  than  72  hours;  the  increased 
resolution  enabled  better  characterization  of  the  analysis  state  and  the  evolution  of  the 
system  over  short  time  scales.  However,  the  deterministic  forecasts  cannot  reveal  the 
effects  of  non-linear  error  growth,  as  stochastic  forecasts  can,  and  hence  deterministic 
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forecasts  become  less  reliable  with  increased  lead  time.  Additionally,  the  ensemble 
forecasts  could  portray  degrees  of  uncertainty  to  users;  this  capability  is  lacking  in 
deterministic  forecast  guidance. 

Zhu  et  al.  also  reinforce  the  idea,  previously  discussed  by  Thompson  (1952),  that 
the  probabilistic  forecast  is  more  useful  in  that  it  allows  users  to  establish  their  own 
decision  criterion  (i.e.,  a  cost-loss  ratio).  Instead  of  being  provided  with  a  deterministic 
forecast  that  suggests  an  “either/or”  decision,  the  forecast  user  can  now  compare  a 
probability  of  occurrence  with  their  own  specific  risk  tolerance;  savvy  users  will  compare 
the  probabilistic  forecast  with  their  own  cost-loss  estimations  to  make  decisions  that  will 
provide  value  over  the  long  term  (Zhu  et  al.  2002). 

Zhu  et  al.’s  and  Thompson’s  findings  have  particular  applicability  to  Department 
of  Defense  (DoD)  weather  forecasters,  who  advise  decision  makers  involved  in 
conducting  operations,  training,  and  managing  resources.  The  ability  of  the  forecaster  to 
accurately  predict  the  most  likely  evolution  of  weather  parameters  and  to  communicate  a 
qualitative  level  of  uncertainty  is  necessary,  but  not  sufficient,  for  optimal  decision 
making.  The  decision  maker  needs  to  know  about  less-likely  but  high-risk  weather 
events;  he  or  she  needs  to  be  able  to  “calculate  the  odds.”  In  many  arenas,  and  often  in 
the  armed  forces,  decision  makers  expect  weather  forecasters  to  give  them  “go/no-go” 
recommendations;  this  leads  the  forecaster  to  combine  the  weather  prediction  with  their 
own  guess  about  the  commander’s  risk  tolerance,  and  to  do  it  implicitly  rather  than 
explicitly.  In  an  attempt  to  clarify  the  concept  of  what  makes  a  “good”  forecast,  Murphy 
(1993)  describes  three  aspects  of  forecast  evaluation:  consistency,  quality,  and  value. 
Consistency  (“type  1  goodness”)  refers  to  how  well  the  forecast  reflects  the  forecaster’s 
best  judgment,  quality  (“type  2  goodness”)  refers  to  how  well  forecast  conditions 
correspond  to  observed  conditions  at  or  around  the  forecast  valid  time,  and  value  (“type  3 
goodness”)  is  detennined  by  whether  the  forecast  yields  some  degree  of  increased  benefit 
to  the  forecast  consumer  (Murphy  1993).  Under  Murphy’s  construct,  many  DoD 
forecasters  are  expected  to  make  an  ex-ante  detennination  of  the  decision  maker’s 
optimum  threshold,  often  at  the  expense  of  consistency.  Probabilistic  forecasts  offer  the 
forecaster  an  opportunity  to  better  meet  the  goal  of  consistency,  while  also  permitting 
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optimization  of  value.  With  full  information,  including  quantitative  uncertainty 
estimates,  decision  makers  can  optimize  decisions  in  a  cost-loss  scenario  as  long  as  the 
cost  of  protective  measures  and  unprotected  loss  can  be  defined.  For  more  complex 
situations,  value  must  be  derived  differently. 

Eckel  et  al.  (2008)  used  two  operational  scenarios  to  demonstrate  the  benefit  of 
employing  stochastic  forecasts  in  conducting  operational  risk  management  (ORM). 
Previous  work  detailed  above  focused  exclusively  on  protection  measures, 
referred  to  as  “defensive  ORM.”  Eckel  et  al.  extended  the  application  of  ORM  to 
offensive  operations,  illustrating  advantages  from  employing  ensemble  forecasting  for 
gains  rather  than  just  for  resource  protection. 

The  authors  used  an  evacuation  simulation,  an  example  of  defensive  ORM,  to 
show  that  over  an  extended  time  frame,  decisions  based  on  stochastic  forecasts  will  result 
in  fewer  cumulative  dollars  spent  for  purposes  of  asset  protection.  In  this  case,  the 
stochastic  operator  yielded  a  30%  savings  rate  over  the  use  of  deterministic  forecasts.  In 
the  offensive  ORM  scenario,  stochastic  forecasting  in  for  destruction  of  enemy  air 
defense  (DEAD)  missions  provided  a  superior  operational  decision  twice  as  often  as  the 
deterministic  solution.  The  stochastic  forecasts  yielded  mission  completion  10%  faster 
than  their  deterministic  forecast  counterparts.  Eckel  et  al.  demonstrated  that  stochastic 
forecasts  can  improve  both  efficiency  and  war-fighter  effectiveness,  in  both  offensive  and 
defensive  ORM  scenarios.  Net  economic  value  is  not  defined  in  the  DEAD  simulation; 
the  quantified  benefit  to  campaign  planners  is  the  earlier  destruction  of  enemy 
emplacements. 

E.  DIRECTION  OF  THIS  STUDY 

This  thesis  will  follow  in  the  direction  suggested  by  Eckel  et  al.  and  Zhu  et  al.  by 
applying  stochastic  forecasting  to  operational  decision-making,  but  extending  the  context 
to  the  more  complex  campaign  scenario  in  WIAT.  It  builds  upon  the  work  done  by  Eckel 
previously,  which  compared  outcomes  for  a  series  of  independent  events.  The  use  of 
WIAT  places  the  ensemble  forecasts  in  the  context  of  a  dynamic  environment  where  each 
forecast  has  direct  impact  on  mission  planning,  the  execution  of  the  mission,  and  hence 

the  necessity  of  related  follow-on  missions.  Simulation  of  more  complex  scenarios, 
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where  use  of  information  from  the  ensemble  forecasting  system  shows  benefit,  should 
help  demonstrate  to  decision  makers  that  information  about  uncertainty,  rather  than  being 
a  nuisance,  has  operational  value. 


16 


III.  THE  WEATHER  IMPACT  ASSESSMENT  TOOL  (WIAT) 


A.  WIAT  AND  SIMULATIONS 

War  games  have  increasingly  become  a  staple  of  military  planning  worldwide 
over  the  last  two  centuries;  the  American  Navy’s  history  of  such  study  extends  over  a 
century,  beginning  with  the  1880s  and  the  early  years  of  the  Naval  War  College. 
Because  battles  in  air,  on  land,  and  at  sea  are  so  expensive  (in  tenns  of  money  and  human 
life)  and  relatively  unique  in  comparison  to  each  other,  operational  planners  have  long 
sought  means  to  try  out  their  strategies  prior  to  employing  them  in  actual  battle.  In  the 
computer  era,  simulations  have  become  an  increasingly  large  aspect  of  not  only 
operational  planning,  but  platform  and  weapon  design  as  well.  Simulations  permit  many 
iterations  of  tests  and  adjustments  using  computer  hardware  and  software  that  are 
considerably  less  costly  than  the  consequences  of  a  poorly  planned  fighter  acquisition  or 
a  faulty  asset  allocation. 

WIAT  is  a  “stochastic  simulation  toolset  used  for  assessing  the  impacts  of 
METOC  forecasts  and  METOC  phenomena  on  strike  operations”  (SPA  2007a).  WIAT 
was  developed  to  help  the  Oceanographer  of  the  Navy  quantitatively  link  improvement 
and/or  degradation  of  forecast  quality  to  improvement  and/or  degradation  of  warfighter 
metrics.  The  program  was  designed  specifically  to  support  analysis  of  a  Strike  Warfare 
(STW)  campaign  scenario;  however,  the  program’s  designers  noted  that  its  methods 
could  be  applied  to  Naval  Special  Warfare  (NSW)  and  Anti-Submarine  Warfare  (ASW) 
missions. 

WIAT  is  programmed  using  the  General  Campaign  Analysis  Model — Core  Tool 

Suite  (GCAM-CTS),  which  is  a  “set  of  object-oriented  model  development  tools  that 

allow  the  military  analyst  to  quickly  and  easily  create  stochastic  simulations”  (SPA 

2007a).  Many  prior  simulation  programs  focused  on  one  aspect  of  campaign  planning; 

GCAM-CTS  was  developed  to  incorporate  system  analysis  (i.e.,  performance  of  a  given 

weapons  platform),  engagement  analysis  (actual  combat  between  given  military  units), 

and  mission  analysis  (success  of  the  assigned  mission)  as  part  of  evaluating  campaign 

strengths,  weaknesses,  and  overall  effectiveness.  The  tool  suite  is  able  to  incorporate 
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data  from  a  variety  of  sources  and  to  provide  measures  of  effectiveness  (MOEs)  that  are 
commonly  understood  across  the  various  armed  forces.  GCAM-CTS  facilitates 
repeatability  so  users  can  test  the  sensitivity  of  various  mission  and  campaign  elements  to 
small  changes  in  conditions;  WIAT  provides  an  interface  that  allows  the  user  to  run  this 
sort  of  analysis  without  any  sophisticated  programming  background  (SPA  2007a). 

B.  THE  CONSTRUCT  OF  THE  WIAT  CAMPAIGN 

The  program  simulates  a  strike  warfare  (STW)  campaign  in  an  operating  area 
covering  Syria,  Jordan,  northern  Saudi  Arabia,  Iraq,  Kuwait,  and  western  Iran.  The  Navy 
task  force  in  WIAT  is  composed  of  a  carrier  air  wing  (CAW),  which  includes  both 
manned  aircraft  and  unmanned  aerial  vehicles  (UAVs),  based  onboard  an  aircraft  carrier 
(CVN)  operating  in  the  eastern  Mediterranean  Sea.  The  wing  is  tasked  with  conducting 
close  air  support  (CAS)  operations  in  support  of  “ground  forces  in  transit  and  during 
engagements,”  as  well  as  strike  (STK)  and  kill  box  interdiction  (KI)  missions  with  targets 
exclusively  within  the  borders  of  Iraq  (SPA  2007a).  The  CAS  mission  details  aircraft  to 
cover  friendly  “Blue”  ground  units  (brigade-level  task  forces  consisting  of  armor  and/or 
mechanized  infantry  units)  as  they  make  a  simulated  move  from  the  northern  and 
southern  borders  of  Iraq  towards  Baghdad.  STK  missions  have  specific  assigned  targets, 
such  as  mobile  ballistic  missile  launchers  or  troop  elements,  while  KI  missions  assign 
aircraft  an  area  inside  Iraq  to  search  for  a  target,  and  engage  it  if  one  is  found.  Every 
mission  is  conducted  by  two  carrier-based  aircraft  of  the  same  type  (i.e.,  manned  and 
unmanned  aircraft  do  not  operate  as  a  mixed  sortie). 

Opposing  “Red”  forces  are  Iraqi  Republican  Guard  (IRG)  divisions,  infrastructure 
elements  (i.e.,  Headquarters  units),  and  mobile  land-based  ballistic  missile  launchers. 
Because  the  campaign  scenario  is  STW-focused,  ground  operations  are  highly  simplified; 
Blue  ground  forces  have  a  fixed  route  and  engage  Red  ground  forces  en  route  to 
Baghdad,  while  Red  IRG  units  attempt  to  protect  Headquarters  elements,  patrol  their 
immediate  areas,  and  engage  Blue  forces. 

Intelligence,  Surveillance,  and  Reconnaissance  (ISR)  is  provided  for  Blue  forces 
by  both  unmanned  and  manned  aircraft,  as  well  as  Field  Air  Commands  (FACs) 

embedded  with  ground  units.  These  units  provide  targeting  infonnation  for  CAS  and 
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Strike  missions,  as  well  as  battle  damage  assessments  (BDA)  for  previously  conducted 
missions.  The  UAVs  in  particular  conduct  the  BDA  mission  on  randomized  routes  over 
the  area  of  responsibility  (AOR). 

C.  WEATHER  CONDITIONS  IN  WIAT 

o  o 

Weather  conditions  in  WIAT  are  modeled  in  a  0.5  x  0.5  grid,  with  a  varying  time 
resolution  (minimum  of  two  hours).  Six  weather  parameters  are  included:  cloud  coverage 
(upper-level,  mid-level,  and  lower-level)  measured  in  octas,  precipitation  in 
millimeters/hour,  visibility  in  meters,  and  wind  speed  in  meters/second.  Weather  is  read 
from  a  Microsoft  Excel  workbook  file,  with  one  worksheet  for  each  time  increment  of  the 
simulation.  Each  worksheet  has  eighteen  grids  representing  a  map  of  the  geographic 
domain:  the  first  set  of  six  grids  holds  the  true  weather  values  for  the  six  weather 
parameters;  the  second  and  third  sets  of  grids  are  designed  for  execution  and  planning 
forecast  values,  which  can  be  input  by  the  user  or  generated  by  WIAT.  Each  map  is  a 
20x34  grid,  with  each  cell  representing  one  half-degree  square,  from  20.508N,  34.474E 
in  the  southwest  to  30.008N,  49.974E  in  the  northeast  (SPA  2007a). 

Data  provided  to  WIAT  can  come  from  a  variety  of  sources,  but  extensive 
instructions  are  provided  with  WIAT  if  the  user  would  like  to  obtain  data  from  the  Air 
Force  Environmental  Scenario  Generator  (ESG).  A  default  weather  file  is  included  with 
program  installation,  covering  the  period  January  1-7,  1993. 

D.  THE  OPERATIONAL  DECISION-MAKING  PROCESS 

The  program  models  the  planning  process  for  hours  to  days  prior  to  mission 
execution,  and  incorporates  weather  conditions  into  operational  decision  making  and 
impacts  through  the  course  of  the  campaign.  Initial  flight  plan  generation  starts  with  the 
Joint  Prioritized  Integrated  Target  List  (JPITL)  and  information  from  ISR  and  FACs  to 
produce  a  Daily  Target  List  and  an  Air  Tasking  Order  (ATO).  This  list  is  used  to 
prioritize  sorties  and  allocate  resources  (manned  and  unmanned  aircraft)  to  the  various 
mission  types.  Once  the  simulation  reaches  a  point  24  hours  before  a  given  mission  is 
scheduled,  the  planning  forecast  is  checked.  If  the  forecast  values  exceed  the  mission 
thresholds,  then  the  mission  is  cancelled  and  replaced;  if  the  forecast  is  below  the 
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threshold,  the  mission  remains  on  the  task  list.  Another  check  is  made  against  the 
execution  forecast  six  hours  prior  to  mission  departure,  and  if  the  forecast  values  are  still 
below  mission  threshold  values.  WIAT’s  inputs  for  campaign  planning  can  be  seen  in 
Figure  4,  the  method  by  which  planning  and  execution  forecasts  is  shown  in  Figure  5, 
and  a  sample  individual  mission  with  weather  impacts  is  shown  in  Figure  6  (SPA  2007a). 

Adaptive  Strike  Mission  Planning  Process 


Flight  Plan  Generation  and  Modification  Timeline 


Multipi&  F /iaht  Plan  Updates  Over  Time  due  to  Forecasting  &  I  SR 


Figure  4.  Depiction  of  Strike  Planning  Process  in  WIAT  -  Flight  Plan  Generation  and 

Modification  (SPA  2007a) 


Adaptive  Strike  Mission  Planning  Process 


Figure  5.  METOC  Forecast  Integration  into  Strike  Warfare  Planning  Cycle  in  WIAT 

(SPA  2007a) 

20 


Sample  Strike  Mission 


Initial 

Planning 


Strike  t  lesion  owtm 
necessary  against  a  specific 
fixed  target  Target  info  rratlcn 
taken  TrcT  JPtTL  A  sea  elated 
weapon  load  -cute  rug  nt  routes 
&  tiring  detemned 


Initial  Mission  Plan 
Generation 


METOC  Input 

Planning  Forecast 


- ► 

Planning 

Forecast 

\ 

> 

Planning  forecast  calls  for 
precipitation  &  overcast  sk.es 
at  low  altitude  METOC 
personnel  reocTTend  TlBsion 
cnangetca  lower  priority 
target-  '.Veapons.  rcuteB.  and 
target  inforration  is  updated 

r 

Aircraft  arrve  at  target  location 
&  weaponsare  dropped  Dueto 
Degraded  conditions  prcBaPlllty 
of  klllfcreecn  weapon  fired  is 
degraded  Tne  aircraft  3re  also 
unaCietc  ooTpiete  BOA  dueto 
low  vlslB  lllty 

V 

/• 

\ 

During 

Mission 

< - 

METOC  Input 

Execution  Forecast 


Mission  Performed 

Actual  Weather 


Execution  forecast  cans  for 
Ted  iut  wind  speeds  and 
Broken -to -overcast  skies  in  tne 
area  of  tne  secondary  target 
Conditions  do  not  Test  MoGo 
standards.  Mission  win  Be 
flown  as  ecneduted 


Execution 

Forecast 


Figure  6.  Sample  mission  progression.  After  "WIAT  Introductory  Briefing,"  1 1  June 

2007. 


E.  MISSION  EXECUTION 

Once  the  mission  departs,  the  forecaster’s  involvement  is  over;  however,  the 
weather  will  impact  the  remainder  of  the  operation.  Aircraft  transit  directly  to  the  target 
or  get  routed  around  bad  weather.  Re-routing  will  result  in  less  time  on  station  and 
potentially  fewer  shots  at  the  target. 

Various  degradation  factors  based  on  weather  impact  the  ability  of  red  and  blue 
forces  to  detect  and  engage  each  other,  while  random  number  draws  determine  the 
outcome  of  each  of  those  events.  Turns  are  simulated  in  five-minute  intervals.  Once  the 
blue  force  has  destroyed  the  red  force,  or  failed  to  engage  within  the  permitted  time  on 
station,  it  returns  to  the  carrier  to  recycle  for  another  mission.  If  the  aircraft  is  shot  down, 
it  is  removed  from  the  simulation.  An  example  of  a  strike  mission  and  the  factors  that 
impact  a  mission’s  success  is  shown  in  Figure  7. 
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Example  Operation 


a* 


Strike  aircraft  takes  off  from  CVN  enroute  to  strike  a  Red  mobile 
land-based  ballistic  missile.  Route  planning  is  updated  every  five 
minutes  based  on  the  weather  at  flight  level  on  the  route. 


Random  Number 

Actual  Blue 
Detection  Probability 

Actual  Red 
Detection  Probability 

User  Probability  of 
Detection  Input 


Relevant  Sensor 
Degradation  - 
Weather 


Random  Number 


User  Probability  of 
Kill  Input 

Relevant  Weapon 
Degradation  - 
Weather 


Resolution  of  Interaction  (Blue  kills  Red,  Blue 
misses  Red,  Blue  shot  down,  etc.) 


Random  Number 


User  Probability  of 
Detection  Input 


Relevant  Sensor 
Degradation  - 
Weather 


Random  Number 


User  Probability  of 
Kill  Input 


Relevant  Weapon 
Degradation  - 
Weather 


Strike  aircraft  arrives  at  the  Red  target’s  location.  The  above  series  of  random  number  draws,  degradation 
values,  probability  of  detection,  and  probability  of  kill  determine  the  outcome.  If  neither  combatant  is  killed, 
and  Blue  has  sufficient  fuel  and  weapons  to  try  again,  the  interaction  repeats  on  the  next  turn  until  fuel  and 
weapons  are  exhausted  or  one  of  the  two  combatants  is  killed. 


—  Strike  aircraft  returns  to  CVN  for  next  mission 

tasking,  based  on  updated  route  planning. 


Figure  7.  Example  of  Operational  Interaction  between  Blue  and  Red  Forces. 

F.  OPTIONS  AND  FEATURES  IN  WIAT 

WIAT  allows  the  user  to  choose  from  among  several  different  methods  of 
simulating  forecast  error;  in  this  thesis,  we  use  Gaussian  errors  for  weather  prediction. 
WIAT  also  allows  customization  of  platform/weapon/sensor  parameters,  environmental 
degradation  factors,  METOC  thresholds,  mission  thresholds,  and  simulation  parameters. 

WIAT  uses  METOC  threshold  parameters  of  Maximum  Yellow  and  Maximum 
Green  to  specify  the  points  at  which  conditions  and  forecasts  move  from  one  color  to 
another  in  the  red/yellow/green  stoplight  paradigm;  red  indicates  meteorological 
conditions  that  would  prevent  safe  operations,  yellow  indicates  borderline  conditions,  and 
green  indicates  safe  conditions.  These  thresholds  are  used  in  calculation  of  METOC 
performance  metrics,  as  well  as  in  the  determination  of  sensor  and  weapon  performance 
degradation. 
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Mission  threshold  parameters  provide  a  go/no-go  value  for  the  simulated  decision 
maker,  and  are  used  determine  when  changes  are  made  in  mission  planning  in  terms  of 
prioritization  and/or  resource  allocation. 

Platform,  weapon,  and  sensor  parameters  allow  the  user  to  tailor  characteristics  of 
each  to  mimic  realistic  capabilities  for  both  Blue  and  Red  forces  (i.e.,  transit  and  loiter 
speeds  of  an  F/A-18,  number  of  weapons  an  aircraft  can  carry,  probability  of  kill  of  a 
given  weapon,  probability  of  detection  of  a  given  sensor,  etc.). 

Environmental  degradation  factors  determine  the  impact  that  weather  conditions 
above  the  specified  METOC  thresholds  will  have  on  the  effectiveness  of  a  weapon  or  a 
detection  capability.  They  work  as  fractional  multipliers  of  sensor  and  weapon 
performance,  for  both  friendly  and  opposing  forces.  The  weather  thresholds  that  affect 
various  sensors  and  weapons  are  shown  in  Table  1.  For  example,  if  visibility,  low  cloud 
cover,  or  mid  cloud  cover  are  over  their  METOC  thresholds,  blue  force  UAV  sensors  will 
have  probability  of  detection  (Pd)  multiplied  by  the  environmental  degradation  factor  (a 
number  between  0  and  1).  Weapons  will  suffer  reduced  probability  of  kill  (Pk)  when 
METOC  thresholds  are  exceeded.  Adverse  conditions  can  also  lead  to  increased  flight 
time  en  route  from  the  CVN  to  the  target,  or  inability  to  conduct  a  battle  damage 
assessment  (BDA)  once  the  engagement  between  Blue  and  Red  forces  has  concluded. 


Weather  Parameters  and  Degradations 


LCD  MCD  HCD  Wind  Speed  Visibility  Precipitation 

BLUE  UAV  Sensors 

Fighter  Sensors 

Ground  Sensors 

Strike  Weapons  (Dumb/GPS-guided  bombs) 
CAS/KI  Weapons  (Laser-guided  bombs) 

XX  X 

XX  X 

X  X 

X 

X  X 

RED  Air  Defense  Sensors 

Ground  Sensors 

X  X 

X  X 

Table  1 .  Weather  parameters  that  affect  Blue  and  Red  platforms  and  sensors  when 

actual  weather  is  over  METOC  thresholds. 

Simulation  parameters  include  features  such  as  the  number  of  simulations  to  run 
in  a  block,  and  the  intervals  at  which  weather  is  displayed  in  the  WIAT  graphical  user 
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interface  (GUI).  One  simulation  parameter  of  particular  interest  to  the  METOC  user  is  the 
ability  to  set  a  given  rate  at  which  mission  cancellation  recommendations  are  accepted  for 
both  planning  and  execution  forecasts.  This  feature  simulates  the  level  of  confidence  the 
decision  maker  holds  in  the  forecasts  provided  by  METOC  personnel  (SPA  2007c). 

G.  WIAT  OUTPUT 

As  the  simulation  is  run  to  the  conclusion  of  the  campaign  period,  WIAT  collects 
various  MOEs  and  provides  metrics  files  that  detail  the  performance  of  Blue  forces  in 
prosecuting  the  campaign.  Three  primary  categories  of  metrics  are  compiled  for  the  user: 
METOC  performance  metrics,  METOC  impact  metrics,  and  operational  impact  metrics. 
METOC  performance  metrics  (also  referred  to  by  the  creators  as  “METOC  metrics”) 
compare  forecast  and  observed  conditions.  Examples  are  forecast  accuracy  (FAC),  false 
alarm  rate  (FAR),  and  weather  event  probability  of  detection  (POD,  which  should  not  be 
confused  with  sensor  Pd). 

METOC  impact  metrics  measure  the  effect  that  weather  forecasts  have  during 
mission  planning.  For  example,  KI  and  Strike  missions  are  either  changed  or  not 
changed  based  on  forecasts.  The  true  weather  at  mission  execution  time  is  compared  to 
the  forecasts  for  that  mission  to  determine  whether  the  forecast-recommended  change 
was  correct  or  not,  or  if  no  change  was  recommended,  whether  that  input  was  correct  or 
not.  If  a  forecast  indicates  adverse  weather  that  results  in  a  mission  change,  and  that 
forecast  is  verified  by  the  true  weather  at  that  later  time  period,  the  mission  is  considered 
“saved.”  In  contrast,  a  forecast  indicating  acceptable  weather  leading  to  no  mission 
change,  but  with  adverse  weather  conditions  occurring  during  the  mission,  would  be 
considered  a  mission  that  “could  have  been  saved.”  Adverse  conditions  in  only  one  of 
the  six  weather  parameters  would  activate  a  mission  change  recommendation,  and 
separate  metrics  are  generated  to  indicate  which  of  the  missions  were  changed  due  to 
adverse  conditions  for  each  weather  parameter.  These  metrics  are  generated  for  all 
execution  and  planning  forecasts  (SPA  2007b). 

It  is  important  to  note  that  in  some  METOC  impact  metrics,  the  CAS  mission  is 
removed  from  consideration  or  absorbed  into  other  mission  categories.  CAS  is 

determined  to  be  the  most  important  mission  of  the  three,  in  that  it  involves  escort  and 
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support  of  ground  forces  in  a  simulation  of  a  ground  campaign  in  Iraq.  This 
determination  yields  the  philosophy  that  if  CAS  is  required  by  ground  forces,  aircraft  will 
always  be  sortied  to  support  that  mission.  Therefore,  while  there  will  be  other  metrics 
associated  with  the  CAS  mission,  METOC  impacts  metrics  are  effectively  eliminated, 
since  forecasters  cannot  recommend  against  the  mission  being  flown. 

Operational  impact  metrics  focus  on  the  impacts  that  the  actual  weather  has  on  the 
conduct  of  the  campaign.  For  example,  the  number  of  times  that  adverse  conditions  led 
to  extended  flight  times  for  aircraft  to  reach  a  given  target  is  reported  as  an  operational 
impact  metric.  This  impact  can  also  cause  decreased  time  over  the  target  due  to  increased 
flight  time  en  route,  inability  to  determine  a  firing  solution,  or  inability  to  conduct  a 
BDA.  Metrics  are  also  collected  to  detennine  Pk. 

H.  LIMITATIONS  TO  WIAT 

There  are  a  series  of  considerations  and  limitations  to  the  simulated  operational 
decision  maker  and  the  campaign  that  are  relevant  to  understanding  how  WIAT  runs  and 
the  metrics  it  generates.  The  parameter  values  themselves  have  some  restrictions.  When 
the  user  chooses  the  Gaussian  distribution  error,  WIAT  takes  the  actual  weather  value  at 
each  point  and  sets  it  as  the  mean  of  the  distribution  from  which  forecasts  will  be 
produced.  The  sigma  value  and  a  randomly  generated  number  provide  the  error  for  that 
given  grid  point,  which  is  then  applied  to  the  actual  value  to  generate  the  forecast  weather 
value.  However,  forecast  values  can  fall  outside  the  realm  of  reality  (i.e.,  cloud  cover 
less  than  0  or  greater  than  8  octas).  In  these  cases,  WIAT  re-assigns  the  value  as  the 
boundary  value;  a  returned  forecast  of  9  octas  would  be  re-assigned  as  8  octas. 
Additionally,  the  Gaussian  distribution  for  the  cloud  cover  parameters  is  discretized — no 
fractional  values  are  permitted  for  forecast  values  (i.e.,  114  octas  of  cloud  cover). 
However,  fractional  values  are  pennitted  for  wind  speed,  visibility,  and  precipitation. 

Several  aspects  of  the  parameter  values  also  either  limit  analysis  or  result  in 
simulation  conditions  that  are  less  than  realistic.  Wind  speed  is  treated  as  one  value  for 
the  surface,  for  the  (unspecified)  altitude  en  route  and  over  target  or  the  kill  box. 


25 


Visibility  in  the  ESG  database  is  a  function  of  precipitation  and  relative  humidity. 
However,  dust  and  aerosols  are  taken  into  account  in  the  setting  of  METOC  thresholds. 
This  mismatch  is  not  transparent  to  the  user. 

There  are  several  situations  in  WIAT  that  do  not  incorporate  realistic  campaign 
planning  concerns.  First,  only  the  weather  at  the  target  is  considered  when  detennining 
whether  or  not  a  mission  will  remain  on  the  ATO  and  take  off  as  scheduled.  The  weather 
en  route  does  impact  the  length  of  flight  for  the  aircraft,  but  cannot  cause  mission 
cancellation  or  the  loss  of  the  aircraft;  the  operational  decision  maker  has  no  means  of 
factoring  an  increased  flight  time  into  a  go/no-go  decision.  Further,  weather  conditions  at 
the  CVN  are  not  incorporated;  in  an  actual  campaign,  weather  at  the  launching  platform 
would  be  an  absolutely  critical  consideration. 

Second,  while  there  is  a  tiered  mission  assignment  process  that  fills  CAS,  Strike, 
and  then  KI  missions,  all  missions  within  those  subsets  are  weighted  exactly  the  same  in 
terms  of  importance.  In  reality,  the  war-fighter  commonly  makes  decisions  between 
whether  to  attack  a  more  important  target  in  adverse  weather  or  a  less  important  target  in 
good  weather.  This  construct  is  never  a  possibility  in  WIAT  -  adverse  weather  over  a 
given  target  will  simply  cause  that  particular  target  to  be  reassigned  to  the  bottom  of  the 
JPITL. 

Aircraft  also  continue  to  loiter  over  the  target  if  they  cannot  sense  it,  or  if  they 
sense  and  engage  the  target  but  do  not  kill  it.  There  is  no  mechanism  for  a  mission  to 
change  its  target  during  flight;  inability  to  sense  the  target  will  force  loiter  time  until  the 
target  can  be  detected,  which  leads  to  situations  where  the  Red  forces  will  have  multiple 
opportunities  to  detect  and  engage  the  Blue  forces. 

A  major  shortfall  of  WIAT  is  that  it  does  not  compute  metrics  for  daily  results. 
Therefore,  there  is  no  method  to  determine  whether  or  not  the  campaign  is  effectively 
completed  more  quickly,  or  whether  or  not  a  given  day’s  events  or  weather  had  a 
proportionately  large  impact  on  metrics  for  the  campaign. 

I.  WIAT  IN  THIS  THESIS 

WIAT  simulates  campaign  events  that  in  the  real  world  are  subject  to  random 

processes,  such  as  whether  or  not  a  sensor  will  detect  a  target,  and  whether  or  not,  once  it 
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is  fired,  a  weapon  will  kill  a  target.  Because  WIAT  uses  random  draws  to  simulate  the 
stochastic  nature  of  these  and  similar  events,  two  simulations  with  identical  user  inputs 
will  not  have  identical  outcomes  or  identical  output  metrics.  To  ensure  the  metrics 
produced  are  representative  results  of  the  input  conditions,  each  simulation  is  run  30 
times,  and  each  metric  is  averaged  over  the  30  runs. 

This  study  focuses  on  the  comparison  between  detenninistic  and  stochastic 
forecasting,  with  changes  in  various  metrics  (instead  of  ROI)  serving  as  an  indication  of 
value.  Changes  were  made  to  a  subset  of  parameters  to  reflect  each  of  the  deterministic 
and  stochastic  scenarios,  which  will  be  detailed  further  in  the  Methodology  chapter.  To 
avoid  confusion,  this  thesis  will  use  the  term  WIAT*  when  referring  to  the  author’s 
application  of  the  program,  while  WIAT  will  refer  to  the  program  in  its  original  context. 
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IV.  METHODOLOGY 


A.  WEATHER  DURING  THE  STUDY  PERIOD 

This  study  focuses  on  the  six  weather  parameters  previously  discussed  for  1-7 
January  1993,  shown  in  Figures  8-14.  The  period  in  question  opens  with  a  high-pressure 
ridge  over  northern  Iran  and  Turkey,  with  a  relatively  weak  area  of  surface  low  pressure 
over  the  eastern  Mediterranean  Sea.  Over  the  next  24-48  hours,  a  small  surface  trough 
moves  southeast  along  the  border  between  Iraq  and  Saudi  Arabia  to  the  Arabian  Gulf. 
The  trough  continues  east  over  the  Zagros  Mountains  of  Iran,  and  orographic  lifting  aids 
in  generating  light  precipitation  over  southeastern  Iraq  and  southwestern  Iran.  The 
pressure  gradient,  which  weakens  with  the  passage  of  the  low-pressure  area  on  4-5 
January,  gradually  re-strengthens  over  6-7  January  as  the  upper  level  ridge  becomes 
more  pronounced.  A  surface  low  also  develops  over  the  eastern  Mediterranean  towards 
the  close  of  the  study  period.  Visibility  is  generally  poor  in  northern  Iraq  throughout  the 
period,  with  periods  of  impacts  to  southern  and  central  Iraq  during  3-4  January  and  again 
on  6-7  January.  Precipitation  was  a  fairly  constant  feature  in  the  eastern  Mediterranean, 
in  the  vicinity  of  the  aircraft  carrier’s  launch  area,  for  the  duration  of  the  period  with  the 
exception  of  5  January.  In  Iraq,  precipitation  was  extremely  sparse  and  sporadic;  the 
mountains  in  northern  Iraq  and  the  vicinity  of  Baghdad  saw  the  heaviest  precipitation 
rates,  just  under  2  mm/hr,  associated  with  the  development  of  the  surface  low  on  7 
January.  Wind  speeds  were  greatest  when  the  surface  ridge-trough  pairing  was  well 
established  and  a  strong  gradient  existed  between  the  two  features. 
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Figure  8.  NCEP/NCAR  Re-analysis  of  Sea  Level  Pressure,  1  January  1993,  with 

noted  features  of  interest. 
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Figure  9.  NCEP/NCAR  Re-analysis  of  Sea  Level  Pressure,  2  January  1993,  with 

noted  features  of  interest. 
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Figure  10.  NCEP/NCAR  Re-analysis  of  Sea  Level  Pressure,  3  January  1993,  with 

noted  features  of  interest. 
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Figure  1 1 .  NCEP/NCAR  Re-analysis  of  Sea  Level  Pressure,  4  January  1993,  with 

noted  features  of  interest. 
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Figure  12.  NCEP/NCAR  Re-analysis  of  Sea  Level  Pressure,  5  January  1993,  with 

noted  features  of  interest. 
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Figure  13.  NCEP/NCAR  Re-analysis  of  Sea  Level  Pressure,  6  January  1993,  with 

noted  features  of  interest. 
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Figure  14.  NCEP/NCAR  Re-analysis  of  Sea  Level  Pressure,  7  January  1993,  with 

noted  features  of  interest. 


B.  MODIFICATIONS  FOR  STOCHASTIC  WIAT* 

In  order  to  isolate  the  impacts  of  deterministic  and  stochastic  forecasts,  inputs 
were  generally  maintained  at  default  WIAT  levels.  For  the  deterministic  forecasts,  the 
only  items  that  differed  from  default  settings  were  the  rates  at  which  the  operational 
decision  maker  accepts  the  METOC  recommendations  (default  levels  were  80%  for  the 
execution  forecast  recommendation  and  50%  for  the  planning  forecast;  for  this  thesis, 
100%  levels  were  employed).  The  user  input  settings  for  WIAT  used  in  the  deterministic 
scenarios  are  detailed  in  Appendix  B.  In  our  study,  mission  thresholds  of  6  octas  cloud 
cover  for  all  three  cloud  levels,  1 1  meters  per  second  wind  speed,  9700  meters  visibility, 
and  10  millimeters  per  hour  of  precipitation  were  used  in  generating  the  probabilistic 
forecasts.  Decision  flow  remains  the  same  in  WIAT*. 

Two  very  significant  changes  are  made  in  the  inputs  for  WIAT  that  enable 

WIAT*  to  simulate  decisions  based  on  stochastic  weather  forecasts  and  probability 

decision  thresholds.  First,  the  weather  prediction  method  shifts  from  using  a  Gaussian 

prediction  method  to  a  hand-entered  forecast.  Instead  of  applying  errors  to  current 
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weather  to  generate  forecast  values  in  WIAT  itself,  the  forecasts  with  appropriate  errors 
are  generated  in  the  Ensemble  Generation  macro;  these  forecasts  are  then  used  to  produce 
the  probability  file  that  serves  as  the  weather  file  for  WIAT.  Second,  the  meaning  of  the 
KI  and  Strike  mission  thresholds  are  changed  from  being  strictly  weather  parameters  (i.e., 
6  octas  of  cloud  cover  or  11  m/s  wind  speed)  to  probability  thresholds.  The  operational 
decision  maker  in  WIAT  is  making  the  go/no-go  decision  based  on  a  comparison  of  the 
likelihood  of  the  weather  at  a  given  location  exceeding  a  METOC  threshold  with  a 
probabilistic  mission  threshold.  Table  2  shows  the  probability  thresholds,  with  their 
equivalent  scaled  inputs  for  WIAT*,  that  were  tested  by  running  30  simulations  for  each 
probability  threshold. 


Probability  Thresholds  and  Corresponding  WIAT  Mission  Threshold  Inputs 


LCD 

MCD 

HCD 

Wind  Speed 

Visibility 

Precipitation 

100.0% 

8 

8 

8 

14.0338 

0.00 

0.5000 

87.5% 

7 

7 

7 

12.2796 

8738.87 

0.4375 

75.0% 

6 

6 

6 

10.5254 

17477.75 

0.3750 

62.5% 

5 

5 

5 

8.7711 

26216.62 

0.3125 

50.0% 

4 

4 

4 

7.0169 

34955.50 

0.2500 

37.5% 

3 

3 

3 

5.2627 

43694.37 

0.1875 

25.0% 

2 

2 

2 

3.5085 

52433.24 

0.1250 

12.5% 

1 

1 

1 

1.7542 

61172.12 

0.0625 

Table  2.  Probability  thresholds  and  their  corresponding  scaled  values  for  input  into 

WIAT*  simulations. 


METOC  performance  metrics  will  no  longer  be  useful  because  WIAT*  would 
attempt  to  verify  each  forecast  individually.  Probabilistic  forecast  verification  must  be 
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against  distributions  of  verifying  observations,  not  individual  observations.  Alternative 
metrics,  in  the  form  of  reliability  diagrams,  are  generated  to  evaluate  the  stochastic 
forecasts;  these  are  presented  in  Chapter  V. 


C.  SIMULATED  ENSEMBLE  AND  STOCHASTIC  FORECASTS 

1.  Determination  of  Appropriate  Sigma  Values 

In  generating  the  simulated  ensemble  forecasts  and  probability  files  for  WIAT,* 
we  needed  to  select  a  set  of  sigma  values.  Because  our  purpose  was  comparison  with 
WIAT  using  deterministic  forecasts,  we  needed  to  choose  the  same  sets  of  values  for  both 
WIAT  and  WIAT*  simulations.  Further,  we  needed  to  ensure  that  our  sets  of  sigma 
values  were  physically  and  operationally  reasonable.  In  the  deterministic  forecasts, 
sigma  represents  the  standard  deviation  of  forecast  error.  In  the  ensemble  forecasts, 
sigma  represents  the  spread  of  the  ensemble,  a  measure  of  forecast  uncertainty. 

In  WIAT,  the  sizes  of  1 -sigma  variations  for  each  of  the  meteorological 
parameters  are  hard-wired  (i.e.,  they  cannot  be  changed  with  user  inputs).  For  example, 
1.0  sigma  in  WIAT  is  equivalent  to  1  octa  of  cloud  cover  and  1.75  meters  per  second  of 
wind  speed.  The  size  of  standard  forecast  errors  when  using  the  Gaussian  error  method 
was  therefore  controlled  by  setting  a  value  for  sigma.  This  value  is  one  of  the  user 
inputs;  however,  there  is  only  one  sigma  value  allowed,  so  the  sizes  of  standard  error  for 
cloud  cover  and  wind  speed  cannot  be  changed  independently.  Table  3  shows 
meteorological  values  for  each  sigma  value  employed. 


MeteorologcdMdanng 

Skjna  Vatue 

LCQIMCQIHCD(Qdas) 

Wind  Speed  (mb) 

Visttfty(m) 

Predpiladon  (rrm/hr) 

0.5 

0.5 

0.875 

0.03125 

1.0 

1.0 

1.750 

0.06250 

2.5 

2.5 

4.375 

21500 

0.15625 

Table  3.  Meteorological  meanings  for  sigma  values  used  based  on  WIAT 

assignment  of  values  for  1 .0  sigma. 
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Research  that  would  establish  appropriate  sigma  ranges  to  use  was  relatively  thin, 
particularly  for  precipitation  and  visibility.  We  found  Sittel’s  cloud  cover  verification 
studies  of  the  Diagnostic  Cloud  Forecast  (DCF)  useful,  however.  CDF  forecasts  cloud 
cover  over  the  continental  United  States  from  a  “statistical  relation  based  on  recent 
performance  of  [a]  mesoscale  model”  (Sittel,  slide  3).  Sittel  finds  that  root  mean  square 
error  (RMSE)  values  averaged  47%  from  forecast  hours  6  to  48  during  June  2009,  with 
little  deviation  from  that  average,  and  no  trend  line  suggesting  better  forecasting  with 
decreased  lead  time.  Sittel  also  describes  a  "20-20  Index,"  a  number  between  0  and  1, 
which  for  values  near  1,  indicates  a  large  number  of  situations  where  the  difference 
between  forecast  and  observation  is  less  than  20%.  The  index  for  June  2009  indicates  that 
there  is  fewer  than  20%  difference  between  forecast/observed  pairs  88%  of  the  time, 
again  remaining  the  same  with  no  appreciable  trend  from  hours  6  to  48. 

Morone  and  Pond  (1994)  studied  the  verification  for  the  Rapid  Update  Cycle 
(RUC)  and  their  analysis  is  instructive  for  establishing  appropriate  sigma  values  for  wind 
speed.  The  RUC  model  is  singular  among  National  Center  for  Environmental  Prediction 
(NCEP)  models  in  that  it  is  used  to  provide  information  on  imminent  weather  conditions; 
it  provides  analysis  and  forecasts  out  to  12  hours  every  three  hours  (Morone  and  Pond 
1994).  The  verification  study  found  that  RMSE  for  model  surface  winds,  as  compared  to 
rawinsonde  observations,  was  between  3  and  4  m/s  at  the  analysis,  3-hour,  and  12-hour 
forecast  times. 

Based  on  these  studies,  as  well  as  limited  ability  to  vary  sigma  values  within 
WIAT  and  WIAT*,  we  chose  sigma  values  of  0.5  and  1.0  for  execution  forecasts  and  1.0 
and  2.5  for  planning  forecasts.  The  larger  sigma  value  at  the  planning  timeframe 
incorporates  the  common  notion  that  the  forecaster  will  generate  better  forecasts  for  the 
execution  forecast  time  frame  than  for  the  planning  forecast. 

2.  Generation  of  Ensemble  Files 

We  created  a  macro  to  generate  ensemble  files  to  be  used  by  WIAT*  in  lieu  of  the 
deterministic  forecast  used  by  WIAT.  A  macro  is  a  tool  within  Microsoft  Excel  that 
allows  direct  coding  (in  the  Visual  Basic  for  Excel  programming  language)  of  desired 

functions  and  references,  instead  of  using  standard  Excel  functions  from  drop-down 
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menus  and  cell  references  respectively.  Direct  coding  allows  Excel  to  run  a  series  of 
computations  without  requiring  continued  input  from  the  user. 

The  macro  reads  actual  weather  data  from  the  input  file  and  creates  forecast  data 
from  it  using  input  sigma  values.  Input  sigma  values  for  the  low  and  high  uncertainty 
cases  were  used  to  generate  the  execution  and  planning  forecast  values  for  30  ensemble 
members.  Corrections  were  applied  to  forecasts  for  values  outside  forecast  boundaries, 
as  described  in  Chapter  II;  a  separate  set  of  ensemble  members  that  did  not  use  the 
“wall”  represented  by  the  maximum  values  for  wind  speed,  visibility,  and  precipitation 
was  also  generated  but  not  run  through  simulations  in  WIAT*.  Graphs  in  the  Results 
Chapter  demonstrate  that  the  forecast  distributions  were  similar,  thus  eliminating  the  need 
to  conduct  the  simulations  in  WIAT*.  The  macro  also  creates  a  probability  file  based  on 
how  many  of  the  30  ensemble  members  exceed  parameter  thresholds  equivalent  to  the 
METOC  thresholds  at  each  location  and  forecast  lead  time.  The  probabilities  in  this  file 
are  then  used  as  forecasts  for  WIAT*. 

Several  alternative  means  of  generating  ensemble  members  were  investigated.  We 
tested  an  ensemble  generation  method  that  included  applying  random  errors  to  the  true 
weather  file,  in  effect  treating  the  ESG  values  as  observed  values,  and  our  new  values  as 
the  true  values.  We  then  used  the  new  truth  values,  applying  random  errors  a  second  time 
to  generate  each  ensemble  member.  Ultimately,  we  decided  against  this  process,  because 
we  wanted  the  errors  of  our  ensemble  members  to  mimic  as  closely  as  possible  the  errors 
of  the  deterministic  forecasts  produced  within  WIAT.  We  also  initially  modeled  a  beta 
distribution  for  the  wind  speed,  visibility,  and  precipitation  errors,  which  would  have 
more  closely  matched  the  typical  distribution  of  these  values.  However,  we  also 
eventually  eliminated  the  use  of  the  beta  distribution  to  more  closely  parallel  WIAT’s 
forecasting  process. 

3.  Distribution  of  Weather  Values 

Weather  input  for  WIAT*  in  stochastic  simulations,  as  discussed  in  Chapter  IV, 
consists  of  scaled  probability  values  for  both  planning  and  execution  forecasts.  Because 
the  forecasts  are  stochastic,  METOC  performance  and  METOC  impact  metrics  are 

irrelevant  for  analysis.  As  a  check  on  the  quality  of  our  code,  after  generating  the  30 
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ensemble  members,  we  compared  the  distribution  of  forecast  values  to  the  distribution  of 
actual  forecast  values.  Then,  we  also  used  reliability  diagrams  to  evaluate  how  well  the 
forecast  probabilities  compared  with  observed  rates  of  occurrence  for  exceeding  the 
METOC  thresholds. 

Figures  15-18  show  binned  distributions  of  actual  weather  values  and  average 
counts  of  forecast  values  for  the  two  forecast  periods,  in  both  low  and  high  uncertainty 
scenarios.  As  is  expected,  the  distribution  of  execution  forecasts  is  generally  closer  than 
the  planning  forecasts  to  the  actual  weather.  Both  forecast  distributions  demonstrate  a 
smoothing  of  peaks,  as  compared  to  the  distribution  of  actual  values,  and  the  smoothing 
increases  as  more  error  is  introduced.  Figures  15  and  16  show  that  low  cloud  cover  has 
peak  values  at  the  boundary  values  of  0  and  8  octas,  with  a  smaller  peak  at  4  octas. 
Higher  levels  of  uncertainty  at  increasing  lead  times  yield  higher  accumulations  at  non¬ 
peak  values. 


Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values  for  LCD: 

Low  Uncertainty  Scenario 
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Figure  15.  Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values 

for  LCD:  Low  Uncertainty  Scenario. 
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Number  of  Actual  Val  ues,  Average  Number  of  Ensemble  Forecast  Values  for  LCD: 
High  Uncertainty  Scenario 
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Figure  16.  Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values 

for  LCD:  High  Uncertainty  Scenario. 


Number  of  Actual  Values.  Average  Number  of  Ensemble  Forecast  Values  for  Wind  Speed: 
Low  Uncertainty  Scenario 
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Figure  17.  Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values 

for  Wind  Speed:  Low  Uncertainty  Scenario. 
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Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values  for  Wind  Speed: 
High  Uncertainty  Scenario 
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Figure  18.  Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values 

for  Wind  Speed:  High  Uncertainty  Scenario. 

Figures  17-18  show  similar  diagrams  for  wind  speed.  In  these  cases,  peak  values 
occur  at  3-4  m/s,  and  more  spreading  of  the  distribution  occurs  as  more  forecast 
uncertainty  is  introduced.  Peaks  in  execution  and  planning  forecasts  also  occur  at  the 
boundary  value  of  zero.  Because  WIAT  adjusts  returned  forecast  values  that  are  lower 
than  0  m/s  to  a  value  of  0  m/s,  the  ensemble  generation  program  used  in  this  study 
follows  the  same  procedure.  As  is  expected,  however,  diminishing  uncertainty  means 
fewer  values  returned  less  than  zero,  and  hence  lower  accumulation  of  values  at  the 
boundary  value. 

An  additional  consideration  was  taken  in  creating  the  scaled  probability  file  that 
serves  as  the  base  weather  file  for  stochastic  runs  in  WIAT*.  WIAT  will  accept 
fractional  octa  values  for  actual  weather  values,  but  will  not  do  so  for  forecast  values:  it 
instead  truncates  the  value  to  an  integer.  For  example,  if  the  actual  weather  value  for  a 
grid  location  is  6.2  octas,  and  applying  the  Gaussian-distributed  error  yields  a  forecast 
value  of  6.8  octas,  WIAT  will  truncate  the  6.8  octas  value  to  6  octas,  artificially  re¬ 
distributing  the  value  below  the  actual  weather  value.  Further,  WIAT  interprets  mission 
threshold  values  as  the  maximum  permissible  value.  In  this  manner,  all  forecast  values 
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between  6  and  6.99...  octas  would  be  returned  to  a  value  of  6  octas,  and  forecast  values 
intended  to  be  over  the  threshold  would  be  interpreted  as  equal  to  the  threshold,  and 
would  not  result  in  a  mission  change. 

To  remedy  this,  an  additional  adjustment  to  forecast  values  from  the  probability 
file  for  each  scenario  was  performed  that  effectively  rounds  the  forecast  value  up  to  the 
next  integer.  The  distributions  of  values  after  this  adjustment  are  shown  in  Figures  19- 
20.  Actual  values  were  not  adjusted,  but  the  chart  demonstrates  how  filtering  true  values 
in  the  same  manner  would  yield  a  different  distribution.  Concentrations  of  0  and  1  would 
vary  significantly  between  actual  and  forecast  files,  but  because  the  values  of  concern  are 
those  from  6  to  8  octas,  this  variation  is  acceptable.  As  it  stands,  forecast  distributions 
above  the  threshold  value  parallel  the  actual  values  very  closely  post-adjustment. 
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Figure  19.  Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values 
for  LCD:  Low  Uncertainty  Scenario,  after  adjustment  to  counter  WIAT  cloud 

cover  value  truncation. 
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Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values  for  LCD: 

High  Uncertainty  Scenario 
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Figure  20.  Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values 
for  LCD:  High  Uncertainty  Scenario,  after  adjustment  to  counter  WIAT  cloud 

cover  value  truncation. 
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V.  COMPARISON  OF  RESULTS  FROM  WIAT  AND  WIAT* 


A.  DETERMINISTIC  WIAT  RESULTS 

1.  METOC  Impact  Metrics 

The  simulation  blocks  for  the  low  and  high  error  scenarios  yielded  METOC 
impact  metrics  that  show  benefit  from  higher  accuracy.  Two  examples  of  the  METOC 
impact  metrics  are  shown  in  Table  4.  As  either  planning  or  execution  sigma  values 
increase,  the  number  of  missions  incorrectly  changed  and  incorrectly  not  changed 
increases.  We  would  expect  that  higher  degrees  of  forecast  error  would  lead  forecasters 
to  unnecessarily  change  the  mission  when  no  change  was  required  (also  known  as  a 
“false  alarm”)  or  allow  the  mission  to  proceed  when  it  should  have  been  changed  (also 
known  as  a  “miss”). 


Selected  METOC  Impact  Metrics  from  Deterministic  Scenarios 


Low  Error 

High  Error 

Average  Number  or  Missions  Incorrectly  Changed 

9.97 

19.13 

Average  Number  of  Missions  Incorrectly  mi  Chanaed 

16.83 

32.60 

Table  4.  Average  number  of  missions  incorrectly  changed  and  incorrectly  not 

changed  for  low  and  high  error  scenarios  in  WIAT. 

2.  Operational  Impact  Metrics 

The  operational  impact  metrics  shown  in  Table  5  do  not  show  as  clear  a  trend  as 
is  evident  in  the  METOC  impact  metrics.  While  STK  and  KI/CAS  kills  per  100  missions 
increase  when  forecasts  incorporate  less  error,  as  should  be  expected,  the  difference 
between  the  two  is  only  on  the  order  of  1-2%.  This  difference  is  statistically  significant 
at  a  95%  confidence  interval  for  STK  kills  per  100  missions  (based  on  one-tailed  t-tests). 
However,  the  difference  in  KI/CAS  kills  per  100  missions  is  not  statistically  significant  at 
that  confidence  interval. 
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Selected  Operational  Metrics  from  Deterministic  Scenarios 


Low  Error 

High  Error 

STKKiils  per  100  Missions 

62.73 

61.62 

KI/CAS  Kills  36f  100  Missions 

26.41 

25.76 

Table  5.  Strike  and  KI/CAS  Kills  per  100  missions  for  low  and  high  error  scenarios 

in  WIAT. 


Most  operational  metrics  did  not  demonstrate  identifiable  trends  for  analysis.  For 
many  of  the  operational  impact  metrics,  the  forecast  weather  does  not  actually  have  a 
direct  impact  on  the  fields  used  to  generate  the  metrics;  often,  the  actual  weather  is  what 
causes  an  impact  to  Blue  forces  trying  to  complete  missions.  For  example,  the  weapons 
probability  of  kill  metric  is  calculated  directly  from  the  number  of  targets  killed  and  the 
number  of  weapons  deployed  against  those  targets.  The  number  of  weapons  deployed  is 
derived  from  whether  or  not  the  Blue  forces  have  detected  and  then  destroyed  the  target, 
both  of  which  are  influenced  by  actual  weather  and  not  a  forecast.  Assuming  that  the 
degradation  factor  due  to  weather  influences  affects  Blue  and  Red  to  the  same  degree, 
even  metrics  such  as  the  number  of  Blue  manned  aircraft  shot  down  cannot  be  attributed 
to  good  or  bad  forecasts,  but  instead  to  the  actual  weather  and  the  capabilities  given  to 
Blue  and  Red  by  the  user. 

B.  STOCHASTIC  WIAT*  RESULTS 

1.  Reliability  Diagrams  and  Calibration 

One  of  the  constant  concerns  for  the  operational  forecaster  is  determining  whether 
or  not  the  forecasts  issued  are  good  portrayals  of  the  environment.  For  stochastic 
forecasts,  the  forecaster  is  concerned  with  reliability  and  resolution.  In  order  to  have 
stochastic  forecasts  that  are  considered  reliable,  the  forecast  probability  of  a  given  event 
should  equal  the  observed  relative  frequency  of  the  event.  Resolution  refers  to  how  well 
the  probabilistic  forecast  distinguishes  between  events  and  non-events. 
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Reliability  diagrams  compare  forecast  probability  with  observed  relative 
frequency.  For  the  stochastic  scenarios  in  WIAT*,  a  macro  was  generated  that  counted 
the  number  of  times  a  given  probability  of  exceeding  the  mission  threshold  was  issued  for 
execution  and  planning  forecasts,  and  compared  it  to  the  count  of  occasions  for 
corresponding  valid  times  and  locations  where  the  actual  weather  exceeded  the  mission 
threshold.  We  would  expect  a  perfectly  reliable  forecast  to  follow  the  straight  line  shown 
in  the  following  charts.  The  introduction  of  random  forecast  error,  as  well  as  some  of  the 
constraints  applied  in  the  formulation  of  the  ensemble  members,  will  yield  forecasts  that 
are  less  than  perfectly  reliable. 

Figures  21-22  show  examples  of  reliability  diagrams  for  the  low  and  high 
uncertainty  scenarios;  remaining  diagrams  can  be  found  in  Appendix  D.  Both 
demonstrate  a  series  of  forecast-observed  rate  pairs  that  reflect  an  over-statement  of  the 
likelihood  of  unlikely  events,  and  an  under-statement  of  the  likelihood  of  likely  events. 
For  example,  Figure  21  depicts  that  when  forecasts  indicate  a  30%  or  lower  likelihood  of 
exceeding  the  mission  threshold,  the  corresponding  actual  weather  value  never  exceeds 
the  threshold;  meanwhile,  for  forecast  probabilities  66.67%  and  higher,  we  see  that  the 
threshold  is  exceeded  in  every  case.  Very  small  numbers  of  forecast  occurrences  away 
from  the  extreme  probabilities  of  0%  and  100%  accentuate  the  divergence  of  the 
ensemble  from  forecasts  with  perfect  reliability.  Error  bars  show  two  standard  deviations 
above  and  below  the  observed  rate  of  occurrence. 
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Figure  2 1 .  Reliability  Diagram  with  Error  Bars  for  LCD  Execution  Forecast,  Low 

Uncertainty  Scenario. 
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Figure  22.  Reliability  Diagram  with  Error  Bars  for  Wind  Speed  Execution  Forecast, 

Low  Uncertainty  Scenario. 
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As  seen  in  Figures  21-22,  our  ensemble  generation  technique  did  not  produce 
reliable  probability  forecasts.  We  made  a  few  attempts  to  change  aspects  of  the  ensemble 
generation  macro  that  we  thought  might  be  responsible  for  the  ensembles  poor 
calibration,  but  were  unable  to  improve  the  ensemble’s  reliability.  In  order  to  correct  the 
forecast  probabilities  of  exceedance,  we  employed  a  calibration  process  as  discussed  in 
Chapter  II.  We  replaced  the  likelihood  derived  from  the  set  of  30  ensemble  members 
with  the  environmental  rate  of  occurrence  associated  with  that  probability.  WIAT*  runs 
for  each  threshold  level  used  for  the  uncalibrated  forecast  were  conducted  for  the 
calibrated  forecasts  as  well. 

2.  Operational  Impact  Metrics 

As  in  the  deterministic  WIAT  runs,  stochastic  WIAT*  metrics  are  relevant  for 
analysis;  unfortunately,  there  are  few  operational  metrics  that  show  definitive  differences 
among  the  various  threshold  levels,  and  many  other  metrics  are  difficult  to  explain. 
Tables  6-9  show  several  operational  metrics  from  WIAT*  runs  at  each  of  the  eight 
mission  threshold  levels,  for  low  and  high  uncertainty,  and  for  uncalibrated  and  calibrated 
probability  forecasts.  Additional  operational  impact  metrics  are  tabulated  in  Appendix  F. 

Comparing  the  metrics  for  the  various  mission  thresholds  used,  we  cannot  identify 
an  optimum  decision  threshold.  Further,  while  it  seems  reasonable  that  using  a  higher 
threshold  (i.e.,  being  willing  to  fly  missions  even  when  there  is  a  very  high  chance  of 
mission  degradation  or  failure  due  to  weather),  could  explain  the  lower  number  of  strike 
kills  per  100  missions,  the  opposite  trend  is  observed  in  the  KI/CAS  kills  per  100 
missions. 

Comparisons  between  the  low  and  high  uncertainty  scenarios  generally  show 
better  strike  kill  rates,  fewer  missions  impacted,  and  fewer  mission  impacts  at  threshold 
levels  of  87.5%,  75%,  and  62.5%,  but  worse  rates  for  the  lowest  three  thresholds. 
KI/CAS  kills  per  100  missions  demonstrates  the  opposite  trend  in  this  comparison  as 
well. 
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Operational  Impact  Metrics  for  Stochastic  Low  Uncertainty  Scenario  (Uncalibrated) 


Metric 

Threshold 

100.0% 

87.5% 

75.0% 

62.5% 

50.0% 

37.5% 

25.0% 

12.5% 

Strike  Kills  per  100  Missions 

65.55 

63.23 

63.11 

63.35 

64.08 

65.66 

68.42 

68.14 

KI/CAS  Kills  per  100  Missions 

28.04 

25.35 

25.65 

25.33 

25.71 

24.77 

24.05 

22.80 

Missions  Impacted 

165.10 

155.07 

154.30 

151.50 

148.73 

142.20 

127.50 

120.20 

Total  Mission  Impacts 

205.40 

179.17 

177.57 

176.37 

171.10 

167.57 

152.80 

144.37 

Table  6.  Operational  impact  metrics  for  WIAT*  runs.  Metrics  are  for  the  low 

uncertainty  scenario,  uncabbrated  probability  forecasts. 


Operational  Impact  Metrics  for  Stochastic  High  Uncertainty  Scenario  (Uncalibrated) 


Metric 

Threshold 

100.0% 

87.5% 

75.0% 

62.5% 

50.0% 

37.5% 

25.0% 

12.5% 

Strike  Kills  per  100  Missions 

65.55 

62.12 

61.73 

62.94 

64.45 

67.07 

68.80 

69.28 

KI/CAS  Kills  per  100  Missions 

28.04 

27.66 

26.92 

25.39 

25.62 

25.03 

23.54 

21.61 

Missions  Impacted 

165.10 

158.67 

156.87 

152.97 

147.63 

131.90 

120.33 

109.13 

Total  Mission  Impacts 

205.40 

190.20 

183.93 

178.50 

172.20 

156.60 

144.70 

129.27 

Table  7.  Operational  impact  metrics  for  WIAT*  runs.  Metrics  are  for  the  high 

uncertainty  scenario,  uncabbrated  probability  forecasts. 


Operational  Impact  Metrics  for  Stochastic  Low  Uncertainty  Scenario  (Calibrated) 


Metric 

Threshold 

100.0% 

87.5% 

75.0% 

62.5% 

50.0% 

37.5% 

25.0% 

12.5% 

Strike  Kills  per  100  Missions 

65.55 

63.16 

63.40 

63.52 

64.18 

65.20 

66.34 

67.04 

KI/CAS  Kills  per  100  Missions 

28.04 

26.49 

26.59 

26.69 

25.27 

24.51 

25.43 

24.39 

Missions  Impacted 

165.10 

155.67 

154.40 

152.50 

148.43 

143.40 

138.50 

130.63 

Total  Mission  Impacts 

205.40 

178.40 

176.37 

177.10 

171.90 

166.03 

160.37 

151.90 

Table  8.  Operational  impact  metrics  for  WIAT*  runs.  Metrics  are  for  the  low 

uncertainty  scenario,  calibrated  probability  forecasts. 
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Operational  Impact  Metrics  for  Stochastic  High  Uncertainty  Scenario  (Calibrated) 


Metric 

Threshold 

100.0% 

87.5% 

75.0% 

62.5% 

50.0% 

37.5% 

25.0% 

12.5% 

Strike  Kills  per  100  Missions 

65.55 

62.99 

63.14 

63.42 

64.14 

65.61 

66.99 

68.32 

KI/CAS  Kills  per  100  Missions 

28.04 

26.09 

25.88 

25.27 

25.97 

25.03 

24.77 

24.25 

Missions  Impacted 

165.10 

156.07 

152.63 

149.93 

147.93 

142.07 

134.13 

126.77 

Total  Mission  Impacts 

205.40 

182.13 

177.23 

176.23 

172.13 

166.87 

155.90 

151.07 

Table  9.  Operational  impact  metrics  for  WIAT*  runs.  Metrics  are  for  the  high 

uncertainty  scenario,  calibrated  probability  forecasts. 


Comparing  metrics  for  simulations  that  used  calibrated  forecasts  to  the  metrics  for 
simulations  that  used  uncalibrated  forecasts,  none  of  the  metrics,  the  number  of  kills  per 
100  missions,  missions  impacted,  or  total  mission  impacts,  showed  a  clear  benefit  to  use 
of  our  calibration  technique.  No  region  of  optimum  mission  threshold  appeared  with 
calibration. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

Results  across  the  spectrum  of  stochastic  WIAT*  runs  may  show  improved 
metrics  for  stochastic  decision-making  and  weather  support.  For  example,  the  number  of 
strike  kills  per  100  missions  for  the  high  error  deterministic  WIAT  runs  was  61.62  (Table 
5)  while  the  strike  kills  per  100  missions  for  the  high  uncertainty  stochastic  WIAT*  runs 
ranges  from  65.55  to  69.28  (Tables  7  and  9).  This  is  an  increase  of  approximately  4-6% 
over  the  range  of  thresholds  tested.  While  this  percentage  may  not  be  a  dramatic 
improvement,  each  additional  success  per  100  missions  can  bring  earlier  air 
dominance/supremacy,  shorter  campaign  lengths,  fewer  expenses  incurred  for  operation 
of  both  man  and  machinery,  and  reduced  chances  of  friendly  forces  being  shot  down. 
Our  somewhat  mixed  results  do  not  allow  us  to  definitively  conclude  that  stochastic 
forecasting  provides  benefit  to  the  simulated  operational  decision  maker  in  this  campaign 
scenario,  but  we  cannot  rule  it  out,  either.  Several  primary  issues  in  WIAT  prevent  us 
from  forming  more  certain  conclusions:  the  construct  of  the  program  itself,  aspects  of  the 
simulation,  and  a  lack  of  full  transparency  as  to  how  metrics  are  accumulated.  However, 
the  limited  improvement  we  did  see  suggests  that  further  study  should  be  pursued,  by 
implementing  different  strategies  for  testing  purposes. 

WIAT  was  designed  primarily  to  allow  users  to  test  the  impacts  of  varying 
degrees  of  deterministic  forecast  quality  in  the  context  of  a  campaign.  Restrictions  and 
adjustments  of  data  in  some  parts  of  the  program  make  perfect  sense  in  light  of  this 
design  aim;  however,  several  features — with  the  requirement  for  cloud  cover  to  be  in 
integer  octa  values  the  prime  example — make  adapting  the  program  for  stochastic 
purposes  extremely  difficult.  The  choice  to  truncate  forecast  values,  while  not 
performing  the  same  function  on  actual  values,  adds  processing  steps  to  the  effort. 
Finally,  hard-coding  of  certain  sigma  values  and  maximum  parameter  values  into  WIAT 
limits  the  range  of  scenarios  where  WIAT  would  be  useful. 
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While  it  would  be  unrealistic  to  expect  the  programmers  of  WIAT  to  have 
generated  a  simulator  on  par  with  video  games  such  as  the  Harpoon  series,  there  are 
multiple  instances  of  modeling  in  WIAT  that  are  less  than  realistic.  In  order  to  parallel 
the  experience,  most  afloat  METOC  division  officers  have  to  deal  with,  it  seems 
reasonable  to  expect  that  weather  conditions  at  the  CVN,  and  not  just  the  target  location, 
should  be  used  in  detennining  whether  a  mission  will  go  as  scheduled.  Weather  en  route 
to  the  target  is  also  a  complexity  that  the  decision  maker  cannot  integrate  into  the 
planning  cycle.  There  is  no  capability  to  prioritize  missions  by  importance,  and  hence  the 
uniform  weighting  by  decision  makers  of  how  to  allocate  their  resources  (i.e.,  to  target 
the  less  important  Red  force  under  good  weather,  or  the  more  important  Red  force  under 
bad  weather)  and  the  different-valued  outcomes  of  these  missions  are  not  factors  in 
WIAT’s  JPITL  and  metrics.  Finally,  while  metrics  are  broken  down  by  execution  and 
planning  forecast  time  frame,  the  success  of  missions  is  not  broken  down  by  day.  There 
is  no  way  to  tell  in  WIAT  whether  or  not  the  majority  of  Red  force  kills,  for  example, 
were  achieved  in  Days  1  through  3  of  the  campaign,  or  evenly  spread  out  over  the  seven 
days.  This  would  be  a  useful  statistic,  given  the  fact  that  there  are  more  missions  over  the 
seven-day  campaign  than  there  are  opportunities  to  fly  them,  and  that  the  user  is  unable  to 
realize  a  shorter  campaign  with  more  accurate  forecasts  (or  even  perfect  detection  and 
weapons  employment).  Further,  some  metrics  seem  to  only  capture  impacts  of  the 
weather  itself,  rather  than  the  forecaster’s  contribution  to  the  decision  maker. 

Because  WIAT  employs  GCAM-CTS  as  part  of  its  package,  the  programming 
behind  the  simulation  is  often  a  bit  obscured.  Were  it  not  for  the  willingness  of  one  of 
the  original  programmers  to  donate  considerable  time  and  effort,  much  would  have 
remained  misunderstood.  Some  elements  of  WIAT’s  programming  are  more  accessible, 
but  if  the  user  wants  to  understand  the  underlying  how’s  and  why’s  of  WIAT 
architecture,  it  requires  extensive  study  of  programming  in  Visual  Basic  for  Applications 
(VBA)  in  Microsoft  Excel.  The  fact  that  VBA  does  not  appear  to  be  widely  used  in  the 
scientific  community  also  limits  the  avenues  for  information  technology  (IT)  support. 

This  particular  weather  period  may  not  have  been  a  good  choice  for  testing  the 
benefits  of  stochastic  forecasting.  As  discussed  in  Chapter  IV,  weather  conditions  during 
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this  period  were  fairly  mild;  three  of  the  six  parameters  studied  were  effectively  non¬ 
factors.  While  cloud  cover  was  present  for  most  of  the  time  frame  at  the  low-  and  mid¬ 
levels,  the  most  challenging  conditions  for  a  strike  campaign  would  have  revolved  around 
the  consistently  poor  weather  in  the  vicinity  of  the  CVN,  which  of  course  was  not 
considered  in  launch  decisions.  A  more  challenging  weather  scenario,  with  extensive 
thunderstorms  and  heavy  rain  or  with  shamal  wind  conditions  may  have  exercised  the 
possibilities  more  thoroughly. 

B.  RECOMMENDATIONS 

This  thesis  prompts  several  recommendations  for  future  study.  WIAT  could 
provide  an  excellent  means  for  further  study  with  several  changes  to  structure  and  output. 
First,  WIAT  should  be  adapted  to  incorporate  stochastic  forecasting  with  its  own  set  of 
inputs  and  metrics.  Reliability  diagrams  would  be  an  appropriate  METOC  performance 
metric  for  use  with  the  stochastic  forecast  method.  Adaptation  would  also  eliminate  the 
need  for  scaling  probability  values,  as  well  as  pennitting  finer  sampling  of  probability 
thresholds  in  search  of  optimal  values.  Incorporating  more  realistic  penalties  and 
complicated  degradation  schema  would  also  enhance  the  suitability  of  the  campaign 
simulation.  While  the  user  ultimately  can  search  out  the  scenario  of  choice,  inclusion  of 
more  challenging  weather  data  files  might  prove  more  useful  to  the  user  than  the  current 
set.  Users  should  also  have  the  capability  to  choose  “simpler”  forecast  simulations  that 
may  only  consider  one  weather  parameter,  run  a  shorter  campaign  length,  or  study  only 
one  mission  type.  Lastly,  allowing  users  to  stipulate  different  sigma  values  for  different 
weather  parameters  would  enhance  the  realism  of  the  error  distribution.  Training  on  the 
program,  which  has  been  offered  by  SPA  in  the  past,  provides  an  extremely  valuable 
introduction  to  the  program. 

Future  study  should  also  attempt  to  find  realistic  ways  to  simulate  the  generation 
of  ensemble  data.  Perturbing  the  actual  parameter  values  to  generate  the  true  weather 
PDF  discussed  in  Chapter  II  was  pursued  earlier  in  the  process  of  this  thesis,  but 
ultimately  was  removed  because  this  constituted  adding  a  source  of  error  where  WIAT 
had  no  comparable  error  introduction.  Results  from  those  forecasts  did  not  yield  more 
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reliable  forecasts  than  those  described  in  this  thesis,  but  a  more  correct  ensemble 
generation  process  should  incorporate  the  notion  that  observed  weather  itself  contains 
error.  The  computing  power  that  would  be  required  to  support  these  sorts  of 
investigations  mandates  the  use  of  a  more  flexible  programming  language;  while 
Microsoft  Excel  and  its  version  of  VBA  have  proven  themselves  useful  in  many  contexts, 
a  program  such  as  MATrix  LABoratory  (MATLAB)  would  likely  prove  far  easier  to  use 
and  more  capable. 

The  most  exciting  research  may  lie  in  adapting  simulations  in  WIAT’s  image  to 
other  warfare  areas.  The  large  search  areas  characteristic  of  ASW,  and  the  relatively 
micro-scale  concerns  typical  of  NSW,  would  each  be  suitable  and  of  interest  for  weather 
impacts  simulations.  Further,  either  of  these  warfare  areas  would  expand  stochastic 
forecasting  studies  to  oceanographic  applications  and  campaigns;  expanding  work  in 
stochastic  forecasting  to  oceanographic  applications  can  only  serve  to  enhance  the 
Navy’s  capabilities  in  the  years  ahead. 
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APPENDIX  A:  STUDY  PERIOD  WEATHER 


A.  SEA  LEVEL  PRESSURE  CHARTS,  1-7  JANUARY  1993 

This  section  shows  NCEP/NCAR  re-analyses  of  sea  level  pressure  for  the  Middle 
East  region  in  general,  and  specifically  the  operating  area  in  the  campaign  scenario  in 
WIAT. 
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Figure  23.  Sea  Level  Pressure,  1  January  1993. 
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Figure  24.  Sea  Level  Pressure,  2  January  1993. 
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Figure  25.  Sea  Level  Pressure,  3  January  1993. 
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Figure  26.  Sea  Level  Pressure,  4  January  1993. 
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Figure  27.  Sea  Level  Pressure,  5  January  1993. 
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Figure  28.  Sea  Level  Pressure,  6  January  1993. 


^^^^TWSSCSS.1^.  V^5Si\  XrSKSJS^SS^SJsS.  N^RJ5S5^. 

xV\V'^  x$*  \V\\^ 


Figure  29.  Sea  Level  Pressure,  7  January  1993. 
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B. 


850-HPA  GEOPOTENTIAL  HEIGHTS,  1-7  JANUARY  1993 


This  section  shows  NCEP/NCAR  re-analyses  of  geopotential  heights  for  850-hPa 
pressure  levels  for  the  Middle  East  region  in  general,  and  specifically  the  operating  area 


in  the  campaign  scenario  in  WIAT. 
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Figure  30.  850  hPa  Geopotential  Heights,  1  January  1993. 
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Figure  31.  850  hPa  Geopotential  Heights,  2  January  1993. 
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Figure  32.  850  hPa  Geopotential  Heights,  3  January  1993. 
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Figure  33.  850  hPa  Geopotential  Heights,  4  January  1993. 
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Figure  34.  850  hPa  Geopotential  Heights,  5  January  1993. 
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Figure  35.  850  hPa  Geopotential  Heights,  6  January  1993. 
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Figure  36.  850  hPa  Geopotential  Heights,  7  January  1993. 
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c. 


500-HPA  GEOPOTENTIAL  HEIGHTS,  1-7  JANUARY  1993 


This  section  shows  NCEP/NCAR  re-analyses  of  geopotential  heights  for  500-hPa 
pressure  levels  for  the  Middle  East  region  in  general,  and  specifically  the  operating  area 
in  the  campaign  scenario  in  WIAT. 
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Figure  37.  500  hPa  Geopotential  Heights,  1  January  1993. 
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Figure  38.  500  hPa  Geopotential  Heights,  2  January  1993. 
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Figure  39.  500  hPa  Geopotential  Heights,  3  January  1993. 
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Figure  40.  500  hPa  Geopotential  Heights,  4  January  1993. 
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Figure  41.  500  hPa  Geopotential  Heights,  5  January  1993. 
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Figure  42.  500  hPa  Geopotential  Heights,  6  January  1993. 
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Figure  43.  500  hPa  Geopotential  Heights,  7  January  1993. 


66 


D.  SURFACE  PRECIPITATION  RATE,  1-7  JANUARY  1993 

This  section  shows  NCEP/NCAR  re-analyses  of  surface  precipitation  rate  for  the 
Middle  East  region  in  general,  and  specifically  the  operating  area  in  the  campaign 
scenario  in  WIAT. 


1/1/93  to  l/l/93 
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Figure  44.  Surface  Precipitation  Rate,  1  January  1993. 
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Figure  46.  Surface  Precipitation  Rate,  3  January  1993. 


Figure  45.  Surface  Precipitation  Rate,  2  January  1993. 
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Figure  47.  Surface  Precipitation  Rate,  4  January  1993. 
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Figure  48.  Surface  Precipitation  Rate,  5  January  1993. 
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Figure  50.  Surface  Precipitation  Rate,  7  January  1993. 


Figure  49.  Surface  Precipitation  Rate,  6  January  1993. 
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SURFACE  RELATIVE  HUMIDITY,  1-7  JANUARY  1993 


This  section  shows  NCEP/NCAR  re-analyses  of  surface  relative  humidity,  the 
primary  factor  in  determining  visibility,  for  the  Middle  East  region  in  general,  and 
specifically  the  operating  area  in  the  campaign  scenario  in  WIAT. 


1/1/93  to  1/1/93 
NCEP/NCAR  Reanalysis 

Figure  51.  Surface  Relative  Humidity,  1  January  1993. 
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Figure  52.  Surface  Relative  Humidity,  2  January  1993. 
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Figure  53.  Surface  Relative  Humidity,  3  January  1993. 
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Figure  54.  Surface  Relative  Flumidity,  4  January  1993. 
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Figure  55.  Surface  Relative  Flumidity,  5  January  1993. 
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Figure  56.  Surface  Relative  Humidity,  6  January  1993. 


Figure  57.  Surface  Relative  Humidity,  7  January  1993. 
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APPENDIX  B:  LIST  OF  DEFAULT  WIAT  SETTINGS  IN 
DETERMINISTIC  SCENARIOS 


This  appendix  lists  the  default  settings  set  to  use  WIAT  in  a  deterministic 
forecasting  scenario. 


Weather  Prediction  Method 

1 

0  -  Perfect  Knowledge 

1  -  Gaussian  Weather  Prediction 

2  -  Time  Shifted  Weather  Prediction 

3  -  Location  Shifted  Weather  Prediction 

4  -  Percent  Accuracy  Weather  Prediction 

5  -  Hand  Entered  Forecast  Prediction 

Weather  Prediction  Parameters 

Short  Term  Parameters 

Sigma  for  Short  Term  Forecast 

0.5/0.75/1.0 

Short  Term  Forecast  Time  Shift 

Not  applicable 

Short  Term  Vertical  Shift 

Not  applicable 

Short  Term  Horizontal  Shift 

Not  applicable 

Short  Term  Green  Percent  Accuracy 

Not  applicable 

Short  Term  Yellow  Percent  Accuracy 

Not  applicable 

Short  Term  Red  Percent  Accuracy 

Not  applicable 

Long  Term  Parameters 

Sigma  for  Long  Term  Forecast 

1.0/  1.5/ 2.0/ 2.5 

Long  Tenn  Forecast  Time  Shift 

Not  applicable 

Long  Term  Vertical  Shift 

Not  applicable 

Long  Term  Horizontal  Shift 

Not  applicable 

Long  Term  Green  Percent  Accuracy 

Not  applicable 

Long  Term  Yellow  Percent  Accuracy 

Not  applicable 

Long  Term  Red  Percent  Accuracy 

Not  applicable 

Blue  Sensor  Environmental  Degradation 

Blue  Ground  Sensor  Environmental  Degradation  Vector 

Green  Visibility  /  Green  Precipitation 

1.0 

Yellow  Visibility  /  Green  Precipitation 

0.9 

Yellow  Visibility  /  Yellow  Precipitation 

0.8 

Red  Visibility  /  Green  Precipitation 

0.7 

Red  Visibility  /  Yellow  Precipitation 

0.6 

Red  Visibility  /  Red  Precipitation 

0.5 

Blue  UAV  Sensor  Environmental  Degradation  Vector 

Green  Visibility  /  Green  Precipitation 

1.0 

75 


Yellow  Visibility  /  Green  Precipitation  0.9 

Y ellow  V isibility  /  Y ellow  Precipitation  0 . 8 

Red  Visibility  /  Green  Precipitation  0.7 

Red  Visibility  /  Yellow  Precipitation  0.6 

Red  Visibility  /  Red  Precipitation  0.5 

Blue  Fighter  Sensor  Environmental  Degradation  Vector 

Green  Visibility  /  Green  Precipitation  1 .0 

Yellow  Visibility  /  Green  Precipitation  0.9 

Y ellow  V isibility  /  Y ellow  Precipitation  0 . 8 

Red  Visibility  /  Green  Precipitation  0.7 

Red  Visibility  /  Yellow  Precipitation  0.6 

Red  Visibility  /  Red  Precipitation  0.5 

Blue  Weapon  Environmental  Degradation 

Type  1  Weapon  Pk  Environmental  Degradation  Vector 

Green/Green  1.0 

Yellow/Green  0.9 

Yellow/Yellow  0.8 

Red/Green  0.7 

Red/Yellow  0.6 

Red/Red  0.5 

Type  2  Weapon  Pk  Environmental  Degradation  Vector 

Green  1.0 

Yellow  0.75 

Red  0.5 

Type  3  Weapon  Pk  Environmental  Degradation  Vector 

Green/Green  1.0 

Yellow/Green  0.9 

Yellow/Yellow  0.8 

Red/Green  0.7 

Red/Yellow  0.6 

Red/Red  0.5 

Red  Environmental  Degradation 

Red  Air  Defense  Sensor  Environmental  Degradation  Vector 

Green/Green  1.0 

Yellow/Green  0.9 

Yellow/Yellow  0.8 

Red/Green  0.7 

Red/Yellow  0.6 

Red/Red  0.5 
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Red  Ground  Sensor  Environmental  Degradation  Vector 


Green/Green 

1.0 

Yellow/Green 

0.9 

Y  ellow/Y  ellow 

0.8 

Red/Green 

0.7 

Red/Yellow 

0.6 

Red/Red 

0.5 

Blue  Weapon  Parameters 

Type  1  Weapon  Max  Range  (nautical  miles) 

15 

Type  2  Weapon  Max  Range 

15 

Type  3  Weapon  Max  Range 

15 

Type  1  Weapon  Max  Pk  per  missile 

0.75 

Type  2  Weapon  Max  Pk  per  missile 

0.75 

Type  3  Weapon  Max  Pk  per  missile 

0.75 

Type  1  Weapon  Single  AC  Load 

2 

Type  2  Weapon  Single  AC  Load 

2 

Type  3  Weapon  Single  AC  Load 

2 

Blue  Fighter  Loiter  Speed  (knots) 

350 

Blue  Fighter  Transit  Speed 

480 

Blue  Fighter  Max  Time  on  Station  (hours) 

4 

Blue  UAV  Flight  Speed 

120 

Blue  Ground  SO  A 

10 

Blue  Ground  SO  A  In  Contact 

5 

Blue  Fighter  Sensor  Pd 

0.9 

Blue  UAV  Sensor  Pd 

0.9 

Blue  Ground  Detection  Pd 

0.9 

Threat  Parameters 

Threat  Parameters  for  Ground  Forces 

Total  Major  Red  Ground  Forces 

5 

Red  Ground  SO  A 

7 

Red  Ground  SO  A  In  Contact 

0 

Red  Ground  Max  Pd 

0.7 

Mobile  Threat  Target  Speed 

8 

Threat  Parameters  for  Air  Defense 

Red  Air  Defense  Weapon  Pk 

0.05 

Red  Air  Defense  Max  Sensor  Range 

15 

Red  Air  Defense  Max  Pd 

0.7 

General  Inputs 

Number  of  Simulations 

30 

Short  Term  METOC  Acceptance  Rate 

1 

Long  Term  METOC  Acceptance  Rate 

1 

Weather  Display  Update  Interval 

6  hours 
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Weather  Threshold  Values 

Max  Green  LCD  3 

Max  Green  MCD  3 

Max  Green  HCD  3 

Max  Green  Wind  Speed  (m/s)  9 

Min  Green  Visibility  Value  (m)  18,300 

Max  Green  Precipitation  Value  (mm/hr)  8 

Max  Yellow  LCD  6 

Max  Yellow  MCD  6 

Max  Yellow  HCD  6 

Max  Yellow  Wind  Speed  (m/s)  1 1 

Min  Yellow  Visibility  Value  (m)  9,700 

Max  Yellow  Precipitation  Value  (mm/hr)  10 

Impact  Thresholds 

Mission  Length  Impact  Threshold  1 .25 

Maximum  Extra  Transit  Time  (hours)  1 

Sensor  Environmental  Degradation  Threshold  Factor  0.5 

BDA  Environmental  Degradation  Threshold  Factor  0.5 

Strike  Mission  Weather  Thresholds 
Short  Term  (Execution) 

Max  Strike  LCD  Short  Term  6 

Max  Strike  MCD  Short  Term  6 

Max  Strike  HCD  Short  Term  6 

Max  Strike  Wind  Speed  Short  Term  (m/s)  1 1 

Min  Strike  Visibility  Short  Term  (m)  9,700 

Max  Strike  Precipitation  Short  Term  (mm/hr)  10 

Long  Term  (Planning) 

Max  Strike  LCD  Long  Term  6 

Max  Strike  MCD  Long  Term  6 

Max  Strike  HCD  Long  Term  6 

Max  Strike  Wind  Speed  Long  Term  (m/s)  1 1 

Min  Strike  Visibility  Long  Term  (m)  9,700 

Max  Strike  Precipitation  Long  Term  (mm/hr)  10 

Kill  Box  Interdiction  (KI)  Mission  Weather  Thresholds 
Short  Term  (Execution) 

Max  KI  LCD  Short  Term  6 

Max  KI  MCD  Short  Term  6 

Max  KI  HCD  Short  Term  6 

Max  KI  Wind  Speed  Short  Term  (m/s)  11 

Min  KI  Visibility  Short  Term  (m)  9,700 

Max  KI  Precipitation  Short  Term  (mm/hr)  10 
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Long  Term  (Planning) 

Max  KI  LCD  Long  Term  6 

Max  KI  MCD  Long  Term  6 

Max  KI  HCD  Long  Term  6 

Max  KI  Wind  Speed  Long  Term  (m/s)  1 1 

Min  KI  Visibility  Long  Term  (m)  9,700 

Max  KI  Precipitation  Long  Term  (mm/hr)  10 
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APPENDIX  C:  LIST  OF  CHANGED  SETTINGS  FOR  WIAT* 
STOCHASTIC  SCENARIOS 


This  appendix  lists  the  settings  in  WIAT  that  are  adjusted  in  order  to  set  the 
scenario  for  WIAT*.  It  also  lists  the  scaled  parameter  values  that  are  employed  as  the 
probabilistic  mission  thresholds  in  WIAT*. 


Weather  Prediction  Method  5 

0  -  Perfect  Knowledge 

1  -  Gaussian  Weather  Prediction 

2  -  Time  Shifted  Weather  Prediction 

3  -  Location  Shifted  Weather  Prediction 

4  -  Percent  Accuracy  Weather  Prediction 

5  -  Hand  Entered  Forecast  Prediction 

Weather  Prediction  Parameters 
Short  Term  Parameters 

Sigma  for  Short  Term  Forecast  Not  applicable 

Long  Tenn  Parameters 

Sigma  for  Long  Term  Forecast  Not  applicable 

Strike  Mission  Weather  Thresholds 
Short  Term  (Execution) 

Max  Strike  LCD  Short  Term 
Max  Strike  MCD  Short  Term 
Max  Strike  HCD  Short  Term 
Max  Strike  Wind  Speed  Short  Term  (m/s) 

Min  Strike  Visibility  Short  Tenn  (m) 

Max  Strike  Precipitation  Short  Term  (mm/hr) 

Long  Term  (Planning) 

Max  Strike  LCD  Long  Tenn 
Max  Strike  MCD  Long  Tenn 
Max  Strike  HCD  Long  Term 
Max  Strike  Wind  Speed  Long  Term  (m/s) 

Min  Strike  Visibility  Long  Term  (m) 

Max  Strike  Precipitation  Long  Tenn  (mm/hr) 

Kill  Box  Interdiction  (KI)  Mission  Weather  Thresholds 
Short  Term  (Execution) 

Max  KI  LCD  Short  Term  As  Shown  in  Chart 

Max  KI  MCD  Short  Term  “ 


As  Shown  in  Chart 
66 

66 

66 

66 

66 

As  Shown  in  Chart 
66 

66 

66 

66 

66 
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Max  KI  HCD  Short  Term 

Max  KI  Wind  Speed  Short  Term  (m/s) 

Min  KI  Visibility  Short  Term  (m) 

Max  KI  Precipitation  Short  Term  (mm/hr) 


Long  Term  (Planning) 

Max  KI  LCD  Long  Term  As  Shown  in  Chart 

Max  KI  MCD  Long  Term 

Max  KI  HCD  Long  Term 

Max  KI  Wind  Speed  Long  Term  (m/s) 

Min  KI  Visibility  Long  Term  (m)  “ 

Max  KI  Precipitation  Long  Term  (mm/hr)  “ 


Probability  Thresholds  and  Corresponding  WIAT  Mission  Threshold  Inputs 


LCD 

MCD 

HCD 

Wind  Speed 

Visibility 

Precipitation 

100.0% 

8 

8 

8 

14.0338 

0.00 

0.5000 

87.5% 

7 

7 

7 

12.2796 

8738.87 

0.4375 

75.0% 

6 

6 

6 

10.5254 

17477.75 

0.3750 

62.5% 

5 

5 

5 

8.7711 

26216.62 

0.3125 

50.0% 

4 

4 

4 

7.0169 

34955.50 

0.2500 

37.5% 

3 

3 

3 

5.2627 

43694.37 

0.1875 

25.0% 

2 

2 

2 

3.5085 

52433.24 

0.1250 

12.5% 

1 

1 

1 

1.7542 

61172.12 

0.0625 

Table  10.  Probability  thresholds  and  their  corresponding  scaled  values  for  input  into 

WIAT*  simulations.  Also  Table  2. 
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APPENDIX  D:  DISTRIBUTION  OF  WEATHER  VALUES 


This  appendix  shows  figures  representing  distribution  of  actual  values,  as  well  as 
average  execution  and  planning  forecast  values  for  the  30  ensemble  members  for  each  of 
the  low  and  high  uncertainty  scenarios.  Additional  figures  are  used  to  demonstrate  the 
effects  of  binning  by  using  a  rounding  function  designed  to  counter  WIAT’s  truncation  of 
forecast  cloud  cover  values. 


Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values  for  LCD: 
Low  Uncertainty  Scenario 
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Figure  58.  Number  of  Actual  Values  and  Average  Number  of  Ensemble  Forecast 
Values  for  LCD:  Low  Uncertainty  Scenario.  Values  represent  the  distribution 
prior  to  applying  the  fix  for  WIAT’s  truncation  of  forecast  values.  Also  Figure 


15. 
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Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values  for  LCD: 

Low  Uncertainty  Scenario 
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Figure  59.  Number  of  Actual  Values  and  Average  Number  of  Ensemble  Forecast 
Values  for  LCD:  Low  Uncertainty  Scenario.  Values  represent  the  distribution 
after  applying  the  fix  for  WIAT’s  truncation  of  forecast  values. 


Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values  for  MCD: 
Low  Uncertainty  Scenario 
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Figure  60.  Number  of  Actual  Values  and  Average  Number  of  Ensemble  Forecast 
Values  for  MCD:  Low  Uncertainty  Scenario.  Values  represent  the  distribution 
prior  to  applying  the  fix  for  WIAT’s  truncation  of  forecast  values. 
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Number  of  Actual,  Average  Number  of  Ensemble  Forecast  Values  for  MCD: 
Low  Uncertainty  Scenario 
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Figure  6 1 .  Number  of  Actual  Values  and  Average  Number  of  Ensemble  Forecast 
Values  for  MCD:  Low  Uncertainty  Scenario.  Values  represent  the  distribution 
after  applying  the  fix  for  WIAT’s  truncation  of  forecast  values. 
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Figure  62.  Number  of  Actual  Values  and  Average  Number  of  Ensemble  Forecast 
Values  for  Wind  Speed:  Low  Uncertainty  Scenario.  Also  Figure  17. 
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Figure  63.  Number  of  Actual  Values  and  Average  Number  of  Ensemble  Forecast 

Values  for  Visibility:  Low  Uncertainty  Scenario. 


87 


Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values  for  LCD: 

High  Uncertainty  Scenario 
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Figure  64.  Number  of  Actual  Values  and  Average  Number  of  Ensemble  Forecast 
Values  for  LCD:  High  Uncertainty  Scenario.  Values  represent  the  distribution 
prior  to  applying  the  fix  for  WIAT’s  truncation  of  forecast  values.  Also  Figure 


16. 


Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values  for  LCD: 

High  Uncertainty  Scenario 
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Figure  65.  Number  of  Actual  Values  and  Average  Number  of  Ensemble  Forecast 
Values  for  LCD:  High  Uncertainty  Scenario.  Values  represent  the  distribution 
after  applying  the  fix  for  WIAT’s  truncation  of  forecast  values. 
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Number  of  Actual  Values,  Average  Number  of  Ensemble  Forecast  Values  for  MCD: 

High  Uncertainty  Scenario 
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Figure  66.  Number  of  Actual  Values  and  Average  Number  of  Ensemble  Forecast 
Values  for  MCD:  High  Uncertainty  Scenario.  Values  represent  the  distribution 
prior  to  applying  the  fix  for  WIAT’s  truncation  of  forecast  values. 
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Figure  67.  Number  of  Actual  Values  and  Average  Number  of  Ensemble  Forecast 
Values  for  MCD:  High  Uncertainty  Scenario.  Values  represent  the  distribution 
after  applying  the  fix  for  WIAT’s  truncation  of  forecast  values. 
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Figure  68.  Number  of  Actual  Values  and  Average  Number  of  Ensemble  Forecast 
Values  for  Wind  Speed:  High  Uncertainty  Scenario.  Also  Figure  18. 
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Figure  69.  Number  of  Actual  Values  and  Average  Number  of  Ensemble  Forecast 
Values  for  Visibility:  High  Uncertainty  Scenario. 
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APPENDIX  E:  RELIABILITY  DIAGRAMS 


This  appendix  shows  reliability  diagrams  for  low-  and  mid-level  cloud  cover, 
wind  speed,  and  visibility  for  each  of  the  low  and  high  uncertainty  scenarios. 
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Figure  70.  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  LCD  Execution  Forecast,  Low  Uncertainty  Scenario.  Also  Figure  21. 
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Figure  71.  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  LCD  Planning  Forecast,  Low  Uncertainty  Scenario. 
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Figure  72.  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  MCD  Execution  Forecast,  Low  Uncertainty  Scenario. 
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Figure  73.  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  MCD  Planning  Forecast,  Low  Uncertainty  Scenario. 
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Figure  74.  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  Wind  Speed  Execution  Forecast,  Low  Uncertainty  Scenario.  Also  Figure 

22. 
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Figure  75.  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  Wind  Speed  Planning  Forecast,  Low  Uncertainty  Scenario. 
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Figure  76. 
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Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  Visibility  Execution  Forecast,  Low  Uncertainty  Scenario. 
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Figure  77.  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  Visibility  Planning  Forecast,  Low  Uncertainty  Scenario. 
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Figure  78.  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  LCD  Execution  Forecast,  High  Uncertainty  Scenario. 
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Figure  79.  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  LCD  Planning  Forecast,  High  Uncertainty  Scenario. 
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Figure  80.  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  MCD  Execution  Forecast,  High  Uncertainty  Scenario. 


104 


o 


T3 

cd 

cd 


o 

£r 


.a 

o 


O- 

Q 

CJ 


%00'OOT 

%Z.9'96 

%££'£6 

%00'06 

%£9'98 

%E£'£8 

%00'08 

%L9'9L 

%££'££ 

%00'OZ. 

%Z.9'99 

%££'£9 

%00'09 

%£9'9S 

%££'£S 

%00'0S 

%Z.9‘9l7 

%E£'£t7 

%00'0t? 

%Z.9'9£ 

%££'££ 

%00'0£ 

%L9'9Z 

%££'££ 

%00'OZ 

%Z.9'9T 

%££'£! 

%00'0T 

%L9'9 

%££'£ 

%00'0 


o 


T3 

cd 

cd 


_aj 

_Q 

£ 

CD 


O 

CD 

OD 


Figure  8 1 .  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  MCD  Planning  Forecast,  High  Uncertainty  Scenario. 
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Figure  82.  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  Wind  Speed  Execution  Forecast,  High  Uncertainty  Scenario. 
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Figure  83.  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  Wind  Speed  Planning  Forecast,  High  Uncertainty  Scenario. 
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Figure  84.  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  Visibility  Execution  Forecast,  High  Uncertainty  Scenario. 
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Figure  85.  Probability  of  Exceeding  Threshold  vs.  Relative  Frequency  of  Occurrence 
for  Visibility  Planning  Forecast,  High  Uncertainty  Scenario. 


109 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


110 


APPENDIX  F:  OPERATIONAL  IMPACT  METRICS  FOR  WIAT* 


This  appendix  shows  average  operational  impact  metrics  for  each  probability 
threshold  in  the  low  and  high  uncertainty  scenarios. 
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